Performance Benchmarks for Qwen Models via RustingStrata

New performance data has emerged regarding the execution of Qwen large language models (LLMs) using RustingStrata, a tool hosted on GitHub. According to a recent technical report shared on Reddit, RustingStrata was benchmarked against the industry-standard llama.cpp across a variety of Qwen model sizes, ranging from 2 billion to 177 billion parameters.

The benchmarks were conducted on a consumer-grade workstation equipped with an NVIDIA RTX 3060 12 GB GPU, a Ryzen 5 7600X processor, and 32 GB of RAM. The tests measured two primary metrics: prefill speed (tokens per second) and decode speed (tokens per second) after a 4K context window.

Comparative Performance Analysis

The results indicate a divergence in performance based on model scale and memory requirements. For smaller, dense models that fit entirely within the GPU’s VRAM, llama.cpp maintained a performance advantage in prefill speeds. For example, the Qwen3.5-2B model achieved a prefill rate of 6,080 tok/s with llama.cpp, compared to 3,859 tok/s with RustingStrata.

However, the data suggests that RustingStrata demonstrates increased efficiency as model sizes exceed the available VRAM. For models such as the Qwen3-Next-80B-A3B-Instruct and the Qwen3.8-Flash-Next (177B), RustingStrata outperformed llama.cpp in both prefill and decode metrics.

Key findings from the unverified benchmark include:

  • Qwen3-Next-80B-A3B-Instruct: RustingStrata recorded a prefill of 1,154 tok/s and a decode of 63.7 tok/s, while llama.cpp recorded 739 tok/s and 33.3 tok/s, respectively.
  • Qwen3.8-Flash-Next (177B): RustingStrata achieved a prefill of 305 tok/s and a decode of 21.3 tok/s, compared to 193 tok/s and 18.1 tok/s for llama.cpp.

The report notes that for models exceeding VRAM capacity, RustingStrata was approximately 1.5x faster in prefill and up to 2x faster in decoding. Additionally, the report mentions that enabling Multi-Token Prediction (MTP) significantly increased performance, with the 35B model reaching approximately 160 tok/s in decoding.

Development Status and Community Feedback

The RustingStrata project is currently in active development. According to the project’s GitHub repository, multi-GPU support is currently in progress but remains subject to technical issues.

Community discussions on Reddit have highlighted several practical considerations for users. Some contributors noted the absence of pre-compiled release binaries on the GitHub releases page, which may present a barrier to entry for users who do not have the Rust programming language environment installed. While the project aims to provide high-performance execution for large-scale models, users have emphasized the importance of accessible, stable releases for broader adoption.