New Quantization Technique Released for Qwen Model Family

Nace, the developer behind the open-source decision model Drex, has announced the release of “KQuant,” a new quantization technique designed to optimize the performance of the Qwen model family on Apple Silicon hardware. According to a technical announcement shared on Reddit by Alish, an ML engineer at Nace, the technique aims to reduce the information loss typically associated with standard 4-bit quantization.

The release focuses on the MLX framework, providing optimized packs for several models, including Qwen 4B, 9B, 27B, 35B, and the Qwen Flash Next 180B. The developer claims that KQuant achieves a significant reduction in KL-divergence—a metric used to measure how closely a quantized model’s predictions match the original full-precision model—compared to existing public 4-bit baselines.

Performance Benchmarks and Technical Methodology

The reported improvements in KL-divergence vary by model size. According to the data provided by Nace, the KQuant 4-bit version of Qwen Flash Next 180B showed a 3.7x reduction in KL-divergence compared to public baselines. Other reported improvements include a 2.6x reduction for the 9B model and a 1.5x reduction for the 27B model.

The technical methodology behind KQuant involves using real-world text to determine optimal scales and offsets for small groups of weights. The process also allocates additional bits to sensitive layers to preserve accuracy. Nace stated that the computational search for these parameters was conducted on cloud GPU clusters using Fable 5.2 and Astra, utilizing 128 subagents.

While these metrics indicate a closer approximation of the original model’s output, it should be noted that KL-divergence measures prediction similarity rather than direct task-accuracy scores. Furthermore, these results have been shared via a self-published announcement on Reddit and have not been independently verified by third-party benchmarks.

Hardware Compatibility and Speed

The KQuant packs are released in standard MLX format, meaning they do not require custom kernels or specialized loaders to run on macOS. Nace recommends using “Kapitsa,” an open-source tool developed by the company, to achieve optimal performance.

In preliminary testing on an M5 Pro chip, Nace reported that KQuant 4-bit models running on the Kapitsa engine were 6% to 15% faster in writing speed compared to standard 4-bit packs from the mlx-community and lmstudio-community.

Availability

The optimized models are currently available for download on Hugging Face. In addition to the 4-bit versions, Nace has released 5-bit (Q5) versions for the 9B, 27B, and 35B models for users requiring higher precision. The company has indicated that GGUF format support is being developed for future release.