Techniques Developed to Run Large Mixture-of-Experts Models on Mobile Devices

Technical Advancements in Local MoE Model Execution

New engineering developments have demonstrated the potential for running large Mixture-of-Experts (MoE) models locally on mobile hardware. According to technical documentation from OpenBMB, researchers have successfully executed a 35B-A3B MoE model on an iPhone 18 Pro Max, achieving a processing speed of 20.51 tokens per second.

The implementation, detailed in an engineering write-up by OpenBMB, utilizes a technique termed Lookahead Expert Scheduling (LES). Unlike traditional methods that may shrink a model or skip specific “experts” to accommodate hardware limitations, LES aims to mitigate the primary bottleneck of mobile AI: the latency involved in moving expert weights from flash storage to the computational unit.

Lookahead Expert Scheduling and Performance Metrics

The LES technique functions by predicting which experts a model will require for subsequent layers. By preloading these weights from flash storage while current computations are still in progress, the system can execute ready experts immediately. This approach, combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, reportedly results in a 6.07× speed increase over a baseline of 3.38 tokens per second.

Data shared via Reddit and GitHub indicates that the model maintains high fidelity despite significant compression. The implementation achieved a 26.7% reduction in expert weight size with a reported 0.16-point drop across five benchmarks. In a low-memory configuration, the system required approximately 3 GB of peak memory. The implementation, which includes the integration of DSpark, has been made open source via the llama.cpp-omni repository on GitHub.

Implementation Challenges and Hardware Constraints

While the performance metrics are notable, technical discussions on community forums suggest that the efficiency of LES relies heavily on the model being heavily quantized. Observers noted that the success of such prefetching techniques is often contingent on whether the device has sufficient RAM to hold all experts or must rely on prefetching from disk.

The ability to hide memory bottlenecks without altering the model’s actual routing logic suggests that this methodology could potentially be applied to other MoE architectures. However, independent verification of these performance benchmarks on consumer-grade hardware remains pending.

Context in the Mobile AI Ecosystem

These developments occur as Apple continues to integrate AI capabilities into its hardware ecosystem. While specific performance benchmarks for Apple’s proprietary “Apple Intelligence” have not been detailed in the same technical capacity as the OpenBMB report, the company’s focus on local AI execution remains a central component of its mobile strategy. As the industry moves toward more complex architectures, the ability to manage high-parameter models on edge devices remains a critical technical hurdle for hardware manufacturers.