Performance Optimizations for Qwen3.8-Flash-Next

New performance benchmarks for the Qwen3.8-Flash-Next model have emerged from independent testing, suggesting significant improvements in decoding speeds for large-scale language models running on consumer-grade hardware. According to a technical report shared on Reddit, a user utilizing a dual RTX 3090 configuration reported a substantial increase in token generation speeds through the implementation of a specific software optimization.

The reported improvements were achieved using llama.cpp and a specific pull request (PR #27861) that introduces a GPU-resident LRU (Least Recently Used) expert cache. This method optimizes Mixture-of-Experts (MoE) models by caching frequently used experts in VRAM rather than storing entire expert layers in host RAM.

Benchmark Results and Technical Implementation

The testing was conducted on a system featuring two RTX 3090 GPUs (PCIe 3.0) and dual Xeon E5-2696 v4 processors, supported by 188 GB of DDR4-2133 LRDIMM RAM. Using the unsloth UD-Q6_K_XL quantization, the user reported the following performance metrics:

  • Baseline Performance: Approximately 17 tokens per second (t/s) decode speed for short and mid-context prompts, and 12 t/s at a 131k context depth.
  • Optimized Performance: 25–29 t/s decode speed for short and mid-context prompts, and approximately 17 t/s at 131k context depth.

The optimization relies on the observation that the experts selected for a specific token are often the same as those used for the preceding dozens of tokens. The user noted a hit rate of 80–85% for coding tasks and a higher rate for prose. To facilitate this, the user reduced the micro-batch size (ubatch) from 2048 to 512, which freed approximately 5 GB of VRAM per GPU, allowing for more cache slots per layer.

While the optimization significantly improves decoding speeds, the user noted that it may result in slower prefill speeds on long prompts.

Community and Comparative Analysis

The results have prompted discussion among hardware enthusiasts and researchers regarding the trade-offs between model intelligence and inference speed. In the Reddit discussion, one participant noted that while the “Flash” version of the model offers higher speeds, the intelligence gap between it and the 27B parameter version may not always justify the performance gains in all use cases.

Other users highlighted the potential for scaling these configurations in professional environments. One commenter mentioned that high-performance computing (HPC) nodes with similar architectures could potentially support large “phalanxes” of Qwen3.8-Flash-Next instances for research purposes.

As of this report, these performance gains remain unverified by independent third-party benchmarks and are based on a specific hardware configuration and software implementation. Further testing on different quantization levels, such as UD-Q4_K_XL, is expected to follow.