Benchmarking and Optimization of Qwen Model Variants

Recent technical evaluations and experimental fine-tunes of the Qwen model series have revealed significant performance variations across different quantization levels and architectural modifications. Independent testing and community-led research have provided data on how these models handle reasoning, coding, and memory efficiency.

Comparative Performance of Qwen 3.8 Variants

Technical benchmarks conducted by independent users on Reddit have compared the performance of Qwen 3.8 Flash Next (IQ4_XS quantization) against the Qwen 3.8 27B FP8 model. According to the reported data, the Flash Next IQ4_XS variant demonstrated superior performance in several key reasoning and mathematical benchmarks.

The Flash Next IQ4_XS model achieved 83.8% on MMLU-Pro, compared to 75.0% for the 27B FP8 version. In the GPQA Diamond benchmark, Flash Next scored 42.5%, while the 27B FP8 model scored 27.5%. For mathematical reasoning, Flash Next recorded 97.5% on GSM8K and 92.5% on German MGSM, outperforming the 27B FP8 model, which scored 90.0% on both.

However, the 27B FP8 model showed a significant advantage in computational efficiency. In a 250-case trial, the 27B FP8 model completed the task in 36 minutes and 27 seconds, whereas the Flash Next IQ4_XS required 2 hours, 5 minutes, and 47 seconds.

Efficiency Gains in Swift-1.5-Qwen3.8-Flash-Next

A specialized fine-tune, identified as Swift-1.5-Qwen3.8-Flash-Next, has been reported to significantly reduce token consumption during reasoning tasks. In an Aider agentic coding benchmark, the Swift-1.5 variant demonstrated a substantial reduction in “overthinking” compared to the base Qwen3.8-Flash-Next model.

The benchmark results indicated that the Swift-1.5 variant used approximately 40% of the median tokens and completed tasks in 40% of the median time of the base model. Despite these reductions in computational overhead, the model maintained comparable quality in coding tasks. For example, in Python, the Swift-1.5 variant achieved a 48.5% first-try pass rate compared to 57.6% for the base model, while in JavaScript, it achieved 41.7% versus 41.7%. The user noted that while the Swift version uses fewer tokens, it showed a slight decrease in the ability to recover on complex tasks requiring a second attempt.

Memory Transfer and Small Language Model Optimization

Experimental research has also focused on improving small language models (SLMs) by transferring memory from larger architectures. A project titled “Qwengram-0.8B” attempted to transfer the n-gram memory from Qwen3.8 Flash-Next into a Qwen3.5-0.8B backbone.

The researcher reported a 5.05% reduction in validation perplexity (PPL), moving from 18.2759 in the stock 0.8B model to 17.3534 in the Qwengram-0.8B version. The architecture utilizes a frozen 51B-parameter PLE memory and a trained reader to inject information into the smaller model. The study noted that while the method improved language model loss, it caused regressions in math and code performance when the reader was trained beyond 15 million tokens.

Fine-tuning and Comparative Accuracy

In a separate evaluation of fine-tuned models, Kev 4B—an Apache-2.0 fine-tune of Qwen3.5-4B—was tested against the Jev model. Using a dataset of 362 recent items, including arXiv papers and GitHub issues, the test found that accuracy levels for both models remained within two points of each other across all tasks. The Jev model demonstrated higher calibration and outperformed Kev 4B in paraphrase detection, scoring 87.0% compared to 74.5%.