Performance Optimization in llama.cpp

A new pull request (PR) submitted to the llama.cpp repository has demonstrated significant performance improvements for specific quantization formats on x86-based CPUs. According to reports circulating on Reddit, the proposed optimization targets the Q2_0 dot product, utilizing x86 VNNI (Vector Neural Network Instructions) and AVX-512 VNNI instead of the existing generic implementation.

The optimization specifically affects the Q2_0 quantization method. While the pull request remains open and has not yet been merged into the main upstream codebase, preliminary benchmarks suggest a throughput increase of approximately 3.0x to 3.6x across various model sizes.

Benchmark Results and Hardware Testing

The reported performance gains were documented using an AMD EPYC 9645 server processor. In controlled tests involving “Bonsai” models ranging from 1.7B to 27B parameters, the throughput improvements were substantial. For an 8B parameter model, decoding speeds reportedly increased from 2.39 tokens per second (tok/s) to 8.20 tok/s.

Other recorded benchmarks include:

  • 1.7B model: 10.22 tok/s increased to 33.28 tok/s (3.26x)
  • 4B model: 4.45 tok/s increased to 13.36 tok/s (3.00x)
  • 27B model: 0.72 tok/s increased to 2.37 tok/s (3.32x)

The developer noted that the 27B baseline was notably slow, with a prompt evaluation pass taking nearly three minutes prior to the optimization.

The implementation also addresses a potential issue on consumer-grade hardware, such as Intel Corp. processors. Testing on an i5-13400 indicated that the Q2_0 path was previously missing the fast path because AVX-512 is fused off on 12th–14th generation Intel chips, despite the presence of AVX-VNNI. In this specific hardware configuration, decoding for a Ternary-Bonsai-8B model reportedly rose from 2.17 tok/s to 6.92 tok/s.

Technical Considerations and Limitations

The optimization is not a universal speedup for all quantization levels; it is strictly limited to the Q2_0 format and does not apply to Q4 or Q5 formats. Furthermore, the technical community has noted that the optimization may result in minor numerical differences due to fused multiply-add behavior. In a smoke test, the optimized version matched the top token of the original version 99.216% of the time.

Experts in the field have clarified that this specific PR will not benefit “IQ2” quantization schemes, as those utilize a different dot product and packing method that cannot reuse the VNNI kernel.

While these results are significant, they remain unverified by independent third-party audits. The benchmarks were conducted in controlled environments, and further testing is required to determine if these performance gains persist on consumer laptops where power and memory-bandwidth limits may impact results.