New Inference Engine Claims High-Speed Execution of Large MoE Models on Consumer Hardware

A new software project, Betelgeuse, is reporting significant performance gains when running large Mixture-of-Experts (MoE) models on consumer-grade hardware. According to developer reports shared on Reddit, the engine—comprising bel-cli and bel-server—enables the execution of the Qwen 3.8 Flash Next model at speeds previously considered unfeasible for standard gaming PCs.

The developer claims that the engine achieves sustained generation speeds of 22.03 tokens per second (tok/s) on a system equipped with an NVIDIA RTX 3060 12GB GPU and 16GB of DDR4 RAM. In specific scenarios where all routed experts are resident in the VRAM or victim pools, the engine reportedly spikes to 48 tok/s.

Technical Architecture and Methodology

The Betelgeuse engine is a fork of the widely used llama.cpp framework. The developer has submitted a pull request to the upstream llama.cpp repository to integrate some of these optimizations.

The reported performance improvements are attributed to several architectural changes designed to manage the high memory demands of MoE models:

  • VRAM Slot Pool and In-Stream CUDA Graph Replay: The engine maintains approximately 5,000 expert slots in VRAM. By using a custom in-stream CUDA kernel, the engine resolves expert hits on-device, allowing the decode pass to run as a single CUDA Graph replay.
  • RAM Victim Cache and io_uring Pager: When an expert is evicted from VRAM, it is moved to a dedicated host RAM victim cache rather than being discarded. This reportedly reduces physical NVMe SSD reads from approximately 71 MB/token to 12.43 MB/token.
  • Data Optimization: To reduce disk bloat, the developer stripped expert payloads from the main GGUF file, separating them into a 35.3 GB experts.bin file.

The developer emphasizes that the engine maintains “100% bit-exact” parity with the stock model, claiming zero degradation in output quality or reasoning compared to the original weights.

Benchmark Comparisons

In comparative testing against standard implementations, Betelgeuse demonstrated substantial speed increases on a single 16GB DDR4 RAM configuration. While stock llama.cpp using mmap produced approximately 1.63 tok/s on a warm run, Betelgeuse achieved 22.03 tok/s.

The impact of memory bandwidth was also highlighted during testing of the IQ4_XS (4-bit) quantization. On a single-channel 16GB RAM setup, the engine reached 14.0 tok/s. When tested in a dual-channel configuration (2x 8GB DDR4), which provides higher host bandwidth, the speed increased to 24.0 tok/s.

Future Development

The developer has indicated that work is underway to port the Betelgeuse architecture to the DeepSeek V4.1 Flash (1017) model. Early internal testing reportedly shows performance around 25 tok/s on similar hardware.

While these benchmarks represent a significant technical achievement for local LLM execution, they are based on self-reported data from a developer on Reddit and have not yet been independently verified by third-party academic or industry benchmarks.