Experimental Conversion of Dense Models to Mixture-of-Experts

An independent developer has reported successful experiments in converting existing dense large language models (LLMs) into sparse Mixture-of-Experts (MoE) architectures without the need for retraining from scratch. According to reports shared on Reddit, the process, referred to as “upcycling,” aims to reduce the computational cost per token by ensuring only a fraction of the model’s MLP (Multi-Layer Perceptron) is active during inference, while maintaining a similar knowledge base.

The experiments were conducted using consumer-grade hardware, specifically an NVIDIA RTX 4060 with 8GB of VRAM. Due to hardware constraints, the developer focused on small-scale models, specifically the Qwen2.5-0.5B and HuggingFaceTB/SmolLM2-360M.

Technical Implementation and Architecture

The converted models utilize a configuration of 32 experts with a “top-8” selection mechanism, supplemented by a shared expert. To maintain stability, certain layers—specifically the final layer and layer 3 of the SmolLM2 model—were retained as original dense layers rather than being converted to MoE. This decision resulted in a slight increase in MLP compute for those specific layers.

The resulting models were exported to the Qwen2-MoE format. The developer noted that while the conversion is possible on limited hardware, the scale of these models limits the immediate performance benefits compared to larger, more resource-intensive models used by major laboratories.

Performance Benchmarks and Limitations

The reported data indicates a significant reduction in MLP compute per token, though this comes at the cost of measurable declines in benchmark accuracy.

Qwen2.5-0.5B Conversion Results:

  • Compute Efficiency: Approximately 40% of the original MLP compute per token.
  • Perplexity (WikiText-2): Increased from 14.66 (dense) to 19.20 (MoE).
  • Mean Accuracy (Benchmarks): Decreased from 0.467 to 0.408 across metrics including HellaSwag, ARC-Easy, ARC-Challenge, WinoGrande, and OpenBookQA.

SmolLM2-360M Conversion Results:

  • Compute Efficiency: Approximately 44% of the original MLP compute per token.
  • Perplexity (WikiText-2): Increased from 12.94 (dense) to 19.11 (MoE).
  • Mean Accuracy (Benchmarks): Decreased from 0.514 to 0.417 across the same benchmark suite.

The developer acknowledged a persistent “quality gap,” noting that because the models utilize at most 50% of their original MLP compute, they cannot currently match the performance of the original dense versions.

Future Outlook

The developer has indicated that future work will focus on converting larger models, implementing longer training periods, and testing more refined expert layouts. While the current results are limited by the compute capacity of consumer-grade GPUs, the experiment serves as a proof of concept for the “upcycling” method in low-resource environments.