Advancements in Local LLM Inference Optimization

Developments in the optimization of Large Language Model (LLM) inference are increasingly focusing on maximizing performance on consumer-grade hardware through low-level software engineering and specialized architectural configurations. Recent technical reports from independent developers highlight significant gains in token generation speeds and memory management, particularly for quantized and ternary models.

Low-Level Optimization for Ternary Models

A developer working on a zero-dependency C inference engine has reported achieving 36.25 tokens per second (tok/s) on a BitNet b1.58-2B-4T model using an Intel Xeon CPU. According to reports on Reddit, the engine is written in pure C99 and avoids heavy runtimes like Python or CUDA.

The performance is attributed to the use of custom AVX2 and AVX-512 routines that allow for direct accumulation into integer registers using VNNI instructions, rather than unpacking weights into float32. The developer noted that while matrix multiplication kernels have been optimized, the system currently faces a “DRAM ceiling,” where decode speed at a batch size of one is limited by memory bandwidth. On the tested Xeon system, the engine is operating at approximately 95% of theoretical memory bandwidth.

Strategies for Large-Scale Mixture-of-Experts (MoE) Models

As model sizes increase, researchers are exploring methods to run massive Mixture-of-Experts (MoE) models on hardware with limited RAM. One developer documented efforts to run a 300B parameter model on a laptop by keeping only the MXFP4-experts in RAM, while streaming the remainder from an SSD.

The findings suggest that read speed is the primary bottleneck for such setups. Several optimization techniques were identified to mitigate this:

  • Sequential Repacking: Converting model weights to a layer-major style to enable large, contiguous sequential reads rather than slower random reads.
  • Speculative Prefetching: Using router hints to predict which experts will be required for the next layer, allowing the system to prefetch data from the SSD during the current layer’s computation.
  • Core Pinning: In some tests, restricting processing to performance cores (P-cores) rather than a mix of P-cores and efficiency cores resulted in a performance increase from 0.29 tok/s to 0.67 tok/s.

Other developers have noted that caching can occasionally degrade performance by causing double-buffering between the SSD and CPU, which consumes significant transfer bandwidth.

Hardware Configuration and Parallelism

In high-end consumer setups, the method of distributing workloads across multiple GPUs remains a critical factor for efficiency. One user proposed a configuration for four RTX 3090 GPUs designed to mimic the performance of professional-grade RTX 6000 cards at a lower cost.

The proposed architecture utilizes a combination of Tensor Parallelism (TP) and Pipeline Parallelism (PP). Specifically, the setup connects two pairs of GPUs via tensor parallelism, with the pairs then feeding into each other via a pipeline. This approach is intended to circumvent the bandwidth limitations of consumer motherboards, as attempting to run all four GPUs in a single tensor parallelism group can result in slower performance due to PCIe lane constraints.