Technical Optimizations in Qwen3.8 Ecosystem

Recent developments in the optimization of the Qwen3.8 model series have highlighted significant performance gains through specialized hardware utilization and modular quantization techniques. While many of these results originate from independent developer testing on platforms such as Reddit and GitHub, they indicate a growing community focus on maximizing inference efficiency across diverse hardware configurations.

High-Performance Benchmarks on Multi-GPU Setups

In a report shared on Reddit, a developer documented significant throughput improvements for the Qwen3.8 27B model using a dual AMD R9700 configuration. By implementing support for MXFP4 (using W4A8 kernels) on top of the Radiance image, the developer reported achieving 280 tokens per second (tok/s) in JSON-formatted tasks.

According to the BetterBench results provided by the developer, the model demonstrated varying performance levels across different categories:

  • JSON: 280.0 tok/s
  • Math: 254.2 tok/s
  • Code: 226.3 tok/s
  • Reasoning: 194.3 tok/s
  • Chat: 148.3 tok/s

The developer noted that the MXFP4 implementation appears to reach the hardware limits of the R9700 cards, surpassing the performance of FP8. Additionally, prefill benchmarks showed a median throughput of 4,894 tokens per second for a 5,918-token prompt, with a Time to First Token (TTFT) of 1.21 seconds. The developer has released the MXFP4 implementation as open-source via Codeberg.

Modular Quantization and N-Gram Integration

Further technical experimentation has focused on “bolting” higher-precision layers onto lower-precision models to balance speed and accuracy. One developer reported testing a method of replacing the 51B N-gram layer in the Qwen3.8 Next model with a Q8 precision version while maintaining an IQ4_XS base.

The unverified test results indicated that this modification did not result in a meaningful speed penalty. In one specific test run on a system equipped with a Xeon e5 2690v4 and an RTX 3090, the token generation speed remained steady at approximately 10.1 to 12.1 tok/s. While the state-dict size increased from approximately 90GB to 115GB, the developer noted that the N-gram layer primarily affects prompt processing rather than token generation.

Hardware Constraints and Model Scaling

The deployment of Qwen3.8 models continues to face challenges regarding memory constraints on consumer-grade hardware. On Apple Silicon, users have reported running the 104GB Qwen3.8-Flash-Next model on a 48GB Mac at approximately 12 tok/s using the Slotstream repository.

Discussions within the developer community also highlight a shift in model release strategies. Following the release of Qwen3.8 27B, some users noted a reduction in the variety of model sizes compared to previous versions, such as the 3.6 series. However, some developers suggest that newer iterations, such as the “Flash Next” models, may serve as technical prototypes to facilitate software optimizations in tools like llama.cpp for future model generations.