Performance Benchmarks in Mixture-of-Experts (MoE) Models

Recent comparative testing of Mixture-of-Experts (MoE) models has highlighted significant disparities in inference speeds between NVIDIA’s Lightning series and Alibaba’s Qwen models. According to unverified performance data shared on Reddit, NVIDIA Lightning 30b demonstrated a notable advantage in task-specific speed, performing approximately 2.5x faster than Qwen3.6 35b on a per-task basis.

In localized testing conducted via llama.cpp on a Strix Halo 395 system, the performance metrics for a 20,000-token prompt were recorded as follows:

  • NVIDIA Lightning 30b: Achieved a prefill speed of 1,059.31 tokens/s and a decode speed of 53.48 tok/s.
  • Qwen3.6 35b (with MTP): Achieved a prefill speed of 988.78 tokens/s and a decode speed of 50.98 tok/s.

While the Lightning model showed superior speed, users noted that the Qwen3.6 35b model exhibited higher levels of intelligence. Observations suggest that Qwen tends to maintain a consistent processing time regardless of the complexity of the query, whereas the Lightning model appears to respond more rapidly when complex reasoning is not required.

Hardware and Memory Architecture Considerations

The efficiency of these large-scale models remains closely tied to underlying hardware architectures and memory management. While NVIDIA has reportedly explored “RAM on die” configurations to optimize performance, current industry trends show a reliance on various memory solutions.

Discussions regarding hardware design suggest that many high-performance devices utilize LPDDR5x soldered directly to the motherboard to avoid the performance and power penalties associated with SODIMM sockets. While NVIDIA has utilized SOCAMM2 sockets in its AI-focused boards to mitigate some of these issues, the industry continues to evaluate the trade-offs between fixed memory and modularity for high-bandwidth AI workloads.

The high margins associated with AI chip production—reportedly reaching 75% gross margins—continue to drive significant interest in the sector. This profitability stands in contrast to traditional gaming chip margins.

As the demand for specialized AI hardware grows, the role of low-level programming remains a subject of debate. While some suggest that AI-driven coding may eventually reduce the need for manual CUDA programming, the ability to understand C++ and CUDA remains a foundational skill for developing the benchmarking tools and performance-verification applications necessary to validate GPU capabilities, such as peak VRAM bandwidth and FLOPs.