Technical Optimization in llama.cpp Targets Intel GPU Performance

A new pull request (PR #26689) submitted to the llama.cpp repository proposes a technical optimization for Intel GPU architectures that could significantly increase inference speeds for large language models (LLMs) using quantized Key-Value (KV) caches. The update focuses on the SYCL backend, specifically adjusting how FlashAttention dispatch decisions are handled on Intel Battlemage hardware.

Technical Implementation and Benchmarks

The proposed change modifies the dispatch gate for quantized-KV decoding. Currently, the system directs these tasks through a “VEC” kernel; the PR introduces a mechanism to switch this path to a “TILE” kernel when quantized KV caches (such as q4_0 or q8_0) are utilized. To allow for testing, the developer has added a new environment variable, GGML_SYCL_FA_DECODE_KERNEL=vec|tile|auto, enabling users to A/B test the performance impact.

According to reports on Reddit, the performance gains are most pronounced as context length increases. On an Intel Battlemage test system, the author reported the following improvements at a 118K context window:

  • Qwen3.6-35B (q4_0 KV): Increased from 12.99 to 29.61 t/s (+127.9%)
  • Qwen3.6-35B (q8_0 KV): Increased from 12.90 to 31.80 t/s (+146.5%)
  • Gemma 4 12B (q4_0 KV): Increased from 5.06 to 13.59 t/s (+168.7%)
  • Gemma 4 12B (q8_0 KV): Increased from 5.13 to 13.81 t/s (+168.7%)

At a lower context threshold of 32K, JIT tests indicated performance increases ranging from 42% to 74% for the tested Qwen and Gemma configurations.

Unverified Results and Hardware Context

While the benchmarks demonstrate substantial potential, the results remain unverified by independent third-party testing. The data is primarily based on author-reported benchmarks and anecdotal testing from community members. Furthermore, the specific Intel GPU SKU used in the primary tests was not specified in the PR, and the optimization specifically targets quantized KV caches; performance for F16 formats remains unchanged under the current dispatch logic.

Independent community testing has provided additional, though similarly unverified, data points. One user reported significant throughput on a B70 GPU, noting that prompt processing (pp) speeds reached 850 tok/s at 42K context and dropped to 560 tok/s at 110K context. In a multi-token prediction (MTP) test involving Qwen3.6 27B, the user reported an aggregate token generation rate of 41.73 tok/s across nine requests.

Market Implications for Intel Hardware

The performance improvements have prompted discussion regarding the value proposition of Intel hardware in the AI space. Some users noted that these optimizations bring Intel’s B70 performance closer to that of competing hardware, such as the AMD R9700, though pricing considerations remain a factor for consumers. As of the current reporting, the PR remains open and has not yet been merged into the main llama.cpp codebase.