Technical Advancements in Local LLM Inference

The Delta Inference Engine has released version 0.93, introducing several performance optimizations and expanded hardware compatibility for running large language models (LLMs) on consumer-grade hardware. According to developer documentation and community reports on Reddit, the update focuses on improving memory management and increasing the speed of token generation when using highly compressed model formats.

Key technical improvements in the v0.93 release include support for 64 GB, 96 GB, and 128 GB RAM configurations. The engine now features a native reader for Engram tables, which allows Unsloth-quantized models to run at full speed. Developers report that the first token latency has been reduced by approximately 40%, with a 180-token prompt responding in roughly 2.3 seconds, down from 3.8 seconds in previous versions.

Performance Benchmarks and Quantization

The update includes significant optimizations for low-bit quantization, specifically regarding the Unsloth IQ2 and IQ3 formats. In testing conducted by the developer on a single NVIDIA RTX 3090 (24 GB VRAM) with 192 GB of system RAM, the UD-IQ2_XXS (2-bit) model achieved a generation speed of 41 tokens per second (tok/s).

In a 99-question benchmark comparing the new UD-IQ2_XXS model against the previous ds4 Q2 version, the new model scored 57 correct answers, compared to 52 in the prior release. This performance is noted to be within two points of the full API model, which scored 59. The developer noted that while these low-bit models are not identical to the full-precision models, they offer significantly higher usability than previous iterations.

The engine also introduces support for the EXL3 format, with a 2.5 bpw (bits per weight) option. Early tests suggest the GPU kernel for this format is approximately twice as fast as previous implementations.

Scaling and Multi-GPU Support

The v0.93 update introduces basic multi-GPU support, which reportedly provides a 7% performance uplift in a configuration utilizing an NVIDIA RTX 3060 and an RTX 3090. However, the developer noted that the engine does not currently scale efficiently across three or more GPUs.

In a separate, unverified report from a user on Reddit, a custom implementation of the DeepSeek-V4.1-Flash model was demonstrated running on eight Tesla V100-SXM2 GPUs. This setup, utilizing a split pipeline and tensor parallelism, reportedly achieved speeds of 215 tok/s for coding tasks and 118 tok/s for legal writing at a 1K prompt length. The user reported that the configuration allowed for a context window of up to 524,288 tokens, with all weights residing in VRAM to avoid system RAM paging.

Hardware Requirements and Compatibility

The Delta Inference Engine is currently optimized for Windows 10 and 11 environments. The technical requirements for optimal performance include:

  • GPU: NVIDIA RTX with at least 12 GB of VRAM (24 GB is recommended for high-speed paths).
  • RAM: A minimum of 64 GB, though 128 GB or more is recommended for larger models.
  • Storage: A fast NVMe SSD is required to handle model files ranging from 233 GB to 395 GB.
  • CPU: Any x86-64 CPU, with AVX2 support providing the fastest performance.

The engine does not currently support AMD GPUs. The project is released under the MIT license and is built on the Strata framework.