A new C++20 implementation of the vLLM serving stack, currently referred to by the developer as vllm.cpp, has been released as an unaffiliated community port. The project aims to address deployment challenges associated with the original vLLM framework, specifically regarding its large installation footprint and reliance on a Python interpreter during inference.

Technical Specifications and Deployment

According to the developer, the primary motivation for the port was to enable the embedding of inference capabilities within other software environments where a Python interpreter is impractical. The original vLLM installation can require approximately 9.1 GiB of virtual environment space; in contrast, the new vllm.cpp compiles to a 66 MiB binary.

The implementation removes the requirement for Python and PyTorch at runtime. It includes several advanced features such as continuous batching, block-paged KV caching, automatic prefix caching, and speculative decoding. The project also provides an OpenAI-compatible server interface. While the developer notes that the project is under heavy development and lacks support for multi-GPU hardware, ROCm, and LoRA at this stage, it does support various architectures including CUDA (sm_80 through sm_121a), CPU (AVX-512 and Arm i8mm), and Metal.

Performance Benchmarks

The developer claims that the engine’s output is checked “token-for-token” against a pinned vLLM oracle to ensure mathematical correctness across approximately 25 architectures.

In unverified performance tests conducted on DGX Spark, Thor, and AGX Orin hardware, the developer reported that vllm.cpp performs nearly identically to the original vLLM under high concurrency. Testing on a Qwen3.6-27B NVFP4 model showed that at a concurrency of 32, vllm.cpp achieved a ratio of 1.017x compared to vLLM.

Regarding memory efficiency, the developer reported that peak GPU usage for the test workload was 40,996 MiB for vllm.cpp, compared to 70,531 MiB for vLLM. However, the developer clarified that this difference is due to a difference in memory allocation strategies—where vLLM pre-reserves a fixed fraction of memory upfront—rather than a more efficient KV cache.

Other reported benchmarks include:

  • Speculative Decoding: The developer claims that using speculative decoding increased throughput from 9.97 to 15.10 tok/s for MTP, and from 10.16 to 29.32 tok/s for DeepSeek-V4-Flash.
  • CPU/Mobile Performance: The engine reportedly achieved 1.18x the prefill speed of llama.cpp on CPU aarch64 and reached 97.6% of the warm total performance of MLX-LM on an M4 chip.

Development Status

The project is released under the Apache 2.0 license and is currently available via GitHub. The developer has noted that the project is in an active state of development, meaning internal structures and available flags are subject to change. While the project offers a stable C ABI for integration, several features—including multimodal support via HTTP API, embedding models, and reranking models—remain unbuilt.