DFlash2 Speculative Decoding Method Released for Large Language Models

Technical Advancements in Speculative Decoding

The landscape of large language model (LLM) inference is seeing significant technical shifts with the release of DFlash2, a speculative decoding implementation designed to accelerate token generation. According to technical reports and developer discussions on platforms such as Reddit, DFlash2 aims to improve the efficiency of model decoding by utilizing a “block drafter” approach.

In one reported implementation involving the Qwen3.8 27B model, DFlash2 demonstrated substantial performance gains. According to a developer report on GitHub, a single-user setup on an NVIDIA RTX 3090 achieved approximately 138 tokens per second (tps) using DFlash2, compared to a baseline of 47.4 tps. This represents a significant increase in generation speed, though the developer noted that performance gains vary depending on the specific task provided to the model.

Performance Benchmarks and Comparative Analysis

The efficacy of DFlash2 appears to be highly dependent on the specific architecture and hardware configuration. In a test conducted by the Atomic.chat team using an NVIDIA RTX 6000, the median results for Qwen3.8 27B showed DFlash2 reaching 140.6 tps, outperforming both the baseline (47.4 tps) and Multi-Token Prediction (MTP) setups (114.7 tps).

However, independent observations suggest limitations in certain contexts. Some users have noted that while DFlash2 provides higher token generation speeds, it may incur a “prefill penalty.” One user reported a drop in prefill speeds from 1500 tok/s to 1300 tok/s when using the official DFlash drafter on an RTX 3090. Additionally, discussions indicate that while DFlash2 excels in low-context scenarios, MTP may remain more efficient for long-context free-form prose or high-concurrency environments (8+ concurrent users).

Optimization and Memory Management

A key component of the recent performance improvements involves the quantization of the drafter. A developer reported that by using a GPTQ int4 quantization for the drafter, memory usage was reduced to approximately 1.19 GB, which significantly improved the efficiency on 24 GB VRAM cards compared to the 3.85 GB required by the bf16 version.

The implementation also introduces “lookup-augmented drafting.” This technique uses a Triton kernel to scan the request’s own token history to propose subsequent tokens, which reportedly increased tokens per step by 29% in tasks involving the reproduction of existing text.

Users have also noted the trade-offs regarding memory allocation. When using DFlash2, some observers reported a higher VRAM requirement compared to enabling MTP, particularly when loading the necessary DFlash models. Furthermore, there are indications that the current GPU kernels may not be as highly optimized for AMD hardware as they are for NVIDIA, according to community feedback.

Implementation and Availability

DFlash2 is currently being integrated into various inference engines. While official support in vLLM remains an unmerged pull request, developers have successfully backported the functionality to version 0.27.1. For users utilizing llama.cpp, the integration of DFlash2 is being tracked via recent pull requests. The technology is currently most effective for single-user to low-concurrency workloads, with specific optimizations available for hybrid models through prefix caching.