Technical Update to Basalt Inference Engine
The Basalt inference engine has released version 0.4.0, introducing support for dense models and expanding its multimodal capabilities. According to developer documentation and community reports on Reddit, the update marks a significant shift for the project, which previously focused on Mixture-of-Experts (MoE) architectures.
Performance Benchmarks and Model Support
The primary feature of the 0.4.0 release is the inclusion of the Qwen 3.8 27B dense model. According to the developer, the engine achieves high-speed throughput on high-end hardware, specifically reporting up to 776 tokens per second (tok/s) for structured output and 262 tok/s for conversational tasks when running on an NVIDIA RTX 5090. Prefill speeds are reported at 12,000 tok/s at a 4k context window, though the developer notes that performance decreases as the context grows.
In terms of accuracy, the developer claims a top-1 accuracy of 92.8% compared to the full BF16 model. The engine also supports a context window of up to 262K tokens and includes capabilities for chat, tools, thinking, images, and video.
Prior to this release, the engine implemented multi-model and router support, including DeepSeek V4.1 Flash. Unverified performance data shared on Reddit suggests that DeepSeek V4.1 Flash (IQ1_M) reached 90 tok/s in conversation and 246 tok/s in structured output on a combination of RTX 5090 and 5060 Ti hardware.
User Feedback and Technical Limitations
While the performance metrics suggest significant speed improvements, community discussions on Reddit highlight several technical challenges and points of contention.
Some users have reported stability issues on Windows operating systems, noting frequent system lock-ups compared to competing software. There are also reports of difficulties loading models on hardware with limited VRAM, such as the RTX 5080, due to the memory requirements of the models.
Furthermore, some technical observers have raised concerns regarding the trade-off between inference speed and model integrity. One commentator noted that while high-speed decoding is achievable, there is a risk of “subtle” degradation in long-horizon, multi-step, or precise tasks when using highly quantized models. There is also ongoing discussion regarding the use of NVFP4 quantization and its impact on fitting large models into consumer-grade VRAM.
Context of Local LLM Development
The update to Basalt arrives amid a broader trend of users moving toward local Large Language Model (LLM) deployment to avoid the “memory crisis” and privacy concerns associated with cloud-based AI services. As developers continue to optimize kernels for new model architectures, the competition between high-speed inference engines and model accuracy remains a central focus for the open-source AI community.
