Technical Advancements in Local LLM Deployment
New developments in the deployment of the Qwen3.8-Flash-Next model have been reported, specifically regarding its integration with the MLX-serve engine. According to a developer on Reddit, a new implementation allows the model to support a context window of up to one million tokens when running on Apple Silicon hardware.
The implementation utilizes a mixed-precision quantization approach, employing 8-bit quantization for dense layers and 4-bit for expert layers. This configuration is designed to maintain high model quality while managing the significant memory requirements of long-context processing.
Performance Benchmarks and Hardware Requirements
Technical data shared by the co-creator of the MLX-serve support indicates that the model can sustain generation speeds of approximately 40 tokens per second for prose and 75 tokens per second for coding at the one-million-token context limit. When tested on an M5 Max with 128GB of memory, prefill speeds were reported between 1,700 and 1,800 tokens per second, with generation speeds remaining near 1,000 tokens per second through the end of the context window.
However, the high memory demand necessitates specific hardware configurations. The developer noted that attempting to run a full one-million-token context requires approximately 117GB of peak memory usage, necessitating the use of the command iogpu.wired_limit_mb=120000 to allocate sufficient resources.
While these benchmarks suggest high efficiency for long-context tasks, users have noted that performance varies significantly based on hardware. For instance, users on community forums have observed that while high-end M-series chips handle these loads, users with lower VRAM capacities may experience significantly slower performance or be limited to smaller context windows, such as 128k or 256k tokens.
Comparative Model Performance
In separate discussions regarding local AI deployment, users have reported that the Qwen3.8-Flash-Next model, specifically in GGUF format, has begun to replace larger models like the Qwen3.8-27B for specific tasks such as coding assistance. One user reported achieving speeds of approximately 30 tokens per second on a personal desktop setup.
The Qwen3.8 series continues to gain traction in the open-source community. Some users have characterized the 27B version as performing at a level comparable to previous frontier-class models, though performance remains highly dependent on the local hardware’s ability to handle the model’s computational load.
The software and model weights for the MLX-serve implementation are currently available via GitHub and Hugging Face, though the developer has cautioned that the engine is still in a stage where bugs may exist and further testing is required for all use cases.