New Inference Engine Targets Edge Computing Performance
A new software engine, BigMoeOnEdge, has been released, aiming to optimize the performance of Mixture-of-Experts (MoE) models on consumer-grade hardware. According to documentation provided by the developer on GitHub, the engine is designed to run any MoE GGUF model compatible with llama.cpp without requiring specialized formatting or repacking.
The engine is designed for cross-platform compatibility, supporting NVIDIA, AMD, and Intel hardware, as well as Apple Silicon, mobile devices, and standard CPUs. It is reportedly compatible with Windows, Linux, macOS, Android, and iOS. A key feature noted by the developer is an --auto command intended to automatically configure settings based on the specific hardware environment.
Performance Benchmarks
In unverified performance benchmarks shared on Reddit, BigMoeOnEdge demonstrated significant speed advantages over existing engines like Strata and colibri, particularly on hardware with limited memory.
On an 8 GB graphics card running the Qwen3.6-35B model, the developer reported a generation speed of 35 tokens per second (tok/s), compared to 23 tok/s for colibri. The report noted that Strata failed to initialize at a 16k context window. In a test involving a long prompt on the same 8 GB card, BigMoeOnEdge reached the first token in 5 seconds, while Strata took 14 seconds and colibri took nearly two minutes.
On a 12 GB card using the same model, BigMoeOnEdge achieved 45 tok/s, while Strata reached 30 tok/s. On a 16 GB MacBook, the engine reportedly delivered 20 tok/s for the Qwen3.6-35B model, compared to 3.5 tok/s for colibri. When testing the larger Qwen3.8-Flash-Next 125B model on the same MacBook, BigMoeOnEdge recorded 10 tok/s, while Strata recorded 7 tok/s.
Technical Limitations and Comparisons
While the engine shows advantages in certain memory-constrained environments, the developer noted that Strata remains slightly faster on hardware where it is specifically tuned. In a test using two 5060 Ti cards with the Qwen3.8-Flash-Next model, Strata was reported to be 1.3x faster than BigMoeOnEdge. The developer attributed this to Strata utilizing custom kernels for specific models, whereas BigMoeOnEdge does not.
The release has prompted discussion among users regarding the importance of “prefill” speeds versus token generation speeds. While the engine offers broad compatibility and ease of use through its automated settings, some users in community discussions have questioned whether the performance gains are consistent across all high-end hardware configurations.
The project is currently hosted on GitHub under the repository Helldez/BigMoeOnEdge.
