New Research Evaluates KV Cache Quantization on Qwen3.8 27B
New technical research has provided a detailed quantitative analysis of how KV cache quantization affects the performance of the Qwen3.8 27B model. The study, shared via a personal blog and discussed on Reddit, utilizes Kullback–Leibler divergence (KLD) to measure how much a model’s output probability distribution shifts when using quantized memory compared to a 16-bit floating-point (bf16) baseline.
The research tested 35 different weight quantizations across three major inference engines: llama.cpp, ExLlamaV3, and vLLM. The testing methodology included a dataset featuring agentic tool-use and various domain-specific texts, with context lengths extending up to 64,000 tokens.
Impact of 8-bit vs. 4-bit Quantization
The findings suggest a significant performance divide between 8-bit and 4-bit KV cache quantization. According to the report, 8-bit KV caches—available in llama.cpp and ExLlamaV3—result in a halving of cache size with “essentially no difference” in KLD. While vLLM showed a slight increase in KLD when using 8-bit FP8 cache, the researcher noted it remains a viable tradeoff for users with limited VRAM.
In contrast, 4-bit KV cache quantization showed measurable performance degradation. The impact was particularly pronounced on higher-precision weights; for instance, on Q8_0 weights, the KLD increased from 0.0007 to 0.0034, a nearly 390% increase. While 4-bit quantization may be a viable option for users running lower-precision weights (Q3 or below), the study generally does not recommend it due to the measurable hit to model accuracy.
Context Length and Error Accumulation
A critical finding of the research is that the negative impact of 4-bit KV cache quantization scales with the length of the input context. In long-document testing, the added KLD was approximately 15% to 30% higher at the 16k–32k token range compared to the initial 4k tokens.
The data indicates that as the number of entries in the KV cache increases, the likelihood of distinct keys becoming indistinguishable due to reduced precision grows. This suggests that for applications requiring very long context windows, such as 128k tokens or more, the degradation from 4-bit quantization could be even more significant.
Engine-Specific Performance Observations
The research also noted variations in how different inference engines handle quantization. vLLM checkpoints were reported to perform worse than smaller GGUF or EXL3 quants, a result the researcher attributed partly to activation quantization.
When running NVIDIA’s NVFP4 and Qwen’s FP8 through vLLM’s Marlin kernels—which maintain activations in bf16—the KLD dropped significantly compared to the standard vLLM checkpoints. Specifically, NVFP4 saw a KLD drop from 0.0256 to 0.0117, while FP8 dropped from 0.0039 to 0.0028.
While these findings provide a technical framework for optimizing model deployment, the results are based on individual research methodology and have not been independently verified by a third-party institutional body.