Technical Optimizations Target Mixture-of-Experts (MoE) Performance Bottlenecks

Developments in the open-source large language model (LLM) ecosystem are increasingly focusing on optimizing Mixture-of-Experts (MoE) architectures, specifically addressing the hardware limitations faced by users running massive models on consumer-grade hardware. Recent technical contributions to the llama.cpp repository and new benchmarks from the Betelgeuse project highlight shifting strategies for managing “out-of-core” MoE execution.

llama.cpp: Addressing VRAM Constraints via CPU Offloading

A significant update to the llama.cpp codebase, specifically the llama_moe_cache pull request by developer am17an, aims to improve performance for users with limited Video RAM (VRAM). The update introduces a mechanism to run “missed experts”—specialized components of an MoE model that do not fit in the GPU memory—on the CPU.

According to reports on Reddit, this optimization is intended to provide a speedup for MoE offloading, allowing larger models to run on hardware with lower VRAM capacity. The update also includes a tuning script to assist in managing these resources. While the implementation aims to bridge the gap between high-end enterprise hardware and consumer setups, users have noted varying experiences regarding multi-GPU stability, particularly when attempting to combine NVIDIA and AMD hardware through Vulkan or dual-backend builds.

Betelgeuse: Mitigating Prefill Bottlenecks in Out-of-Core MoEs

Parallel to these developments, the Betelgeuse project has released performance data regarding the execution of the DeepSeek V4.1 Flash model on limited hardware. A primary challenge in running large MoE models from storage (SSD) or system RAM is the “Time To First Token” (TTFT), often caused by the “prefill” stage where the model processes the initial prompt.

In standard implementations, the prefill stage can be slow because a prompt often activates nearly every expert across all model layers, causing a bottleneck as data moves across the PCIe bus. According to documentation provided by the Betelgeuse developers, the project addresses this through three specific technical changes:

  1. Location-Invariant Cache Partitioning: Maintaining resident slots in both VRAM and Host-RAM to route data without interference.
  2. Single-Pass Layer-Major Prefill: Utilizing 16-token micro-batches to maintain higher Tensor Core occupancy.
  3. Physical VRAM Consolidation: Dynamically swapping the most active experts into VRAM at the end of the first pass.

In unverified benchmarks shared on Reddit, these optimizations allowed a DeepSeek V4.1 Flash Q8_XL model to achieve a warm prompt processing speed of 61.80 tok/s and a decode speed of approximately 18–19 tok/s on an NVIDIA RTX 3060 (12GB) and 16GB of DDR4 RAM. These figures represent a significant increase over the 3–6 tok/s reported in standard offloading configurations.

Comparative Performance and Implementation

The Betelgeuse benchmarks suggest that once the cache is “warmed” through initial processing, disk I/O requirements drop significantly, allowing for sustained generation speeds. However, some community members have noted that these performance gains may be partially attributed to the reuse of previously loaded experts during a session, which reduces subsequent disk reads.

As MoE models continue to scale in parameter count, the industry’s focus is shifting toward these specialized memory-tiering and caching strategies to make frontier-class models accessible on consumer-grade silicon.