Performance Discrepancies in Strix Halo LLM Inference

Technical discussions within the local large language model (LLM) community have highlighted significant performance gaps in running inference on ADVANCED MICRO DEVICES (AMD) Strix Halo hardware. According to reports circulating on Reddit, the official implementation of llama.cpp is reportedly failing to utilize the hardware’s full potential, with users observing that it struggles to reach 50% of the device’s theoretical performance.

In specific tests involving the Qwen 3.8 Flash Next (Q38FN) model, the official llama.cpp implementation reportedly achieved a decode rate of approximately 2 tokens per second (t/s) and a prefill rate of 200 t/s.

Alternative Software Forks and Benchmarks

Several community-developed forks on GitHub have emerged, claiming to offer significantly higher throughput by optimizing for the Strix Halo architecture (gfx1150). These results, however, remain unverified by independent third-party benchmarks.

Reported performance metrics from these community projects include:

  • Halogen-Flash-Server: Claims to reach approximately 50 t/s for decoding and 1200 t/s for prefill, representing roughly 90% of theoretical hardware capacity.
  • Experimental llama.cpp Fork (myhacsint): Reports nearly 60 t/s for decoding and 600 t/s for prefill, cited as roughly 80% of theoretical capacity.
  • Strix-llama.cpp (halo-box): Reports approximately 30 t/s for decoding and 800 t/s for prefill, cited as roughly 75% of theoretical capacity.

While these figures suggest a substantial increase in efficiency, some developers have expressed caution regarding the trade-off between speed and model quality. There are concerns that highly optimized inference engines may inadvertently degrade perplexity or other quality metrics to achieve higher token rates.

Development Roadmap and Technical Challenges

The official llama.cpp project is reportedly working on performance improvements. Current observations indicate that ROCm-based implementations have reached prefill rates of approximately 500 t/s using Qwen 3.8 Flash Next (Q4_K_XL), while Vulkan-based implementations have reached 400 t/s. Decoding rates on ROCm are noted at approximately 20 t/s in early context stages.

Technical hurdles cited by developers include the lack of sparse attention support, which impacts performance as context grows, and the absence of Multi-Token Prediction (MTP) and Grouped Query Attention (GQA) optimizations, which are viewed as essential for reaching higher generation rates on this hardware.

Software Ecosystem and Developer Support

The development of high-performance AI software for AMD hardware continues to face challenges regarding documentation and developer accessibility. Some developers have noted that compared to NVIDIA, AMD’s documentation can be limited, making it difficult to implement advanced features or resolve specific technical issues.

In the broader GPU market, discussions among users suggest a growing divide between consumer-grade hardware and the high-cost silicon required for AI investments. Some users have expressed frustration with the pricing of high-end NVIDIA components, suggesting that the focus on AI-driven hardware may be shifting market dynamics away from the general consumer.