New Inference Engine Enables Large-Scale MoE Models on Single Mini PCs
Developers have announced the release of a new inference engine, “Kyojin,” designed to run massive Mixture-of-Experts (MoE) models on consumer-grade hardware. According to reports from Reddit and GitHub, the engine—built on top of ExLlamaV3—is optimized for the AMD Strix Halo architecture (gfx1151) using the open ROCm framework.
The release demonstrates the ability to run two distinct ~300B-class MoE models on a single 128 GB mini PC powered by the Ryzen AI Max+ 395. The models featured in this initial release are GLM-5.3-Flash and MiMo-V2.6-Flash-MOPD.
Performance Benchmarks and Quantization
The developers provided specific performance metrics for the models, though these figures have not yet been independently verified.
The GLM-5.3-Flash model, which occupies approximately 99.7 GB of memory, achieved a prefill speed of roughly 580 tokens per second (tok/s) at a 3.5K context window, dropping to 546 tok/s at 64K. Its decoding speed was measured between 26 and 30 tok/s using Multi-Token Prediction (MTP).
The MiMo-V2.6-Flash-MOPD model, with a footprint of 105 GB, showed a prefill rate of approximately 650 tok/s at a 4K context window. Decoding speeds varied by task: 32 tok/s for prose, 35 tok/s for chat, and up to 44 tok/s for code when utilizing speculative decoding, while plain text decoding averaged 29 tok/s.
To fit these massive models into 128 GB of memory, the developers utilized advanced quantization techniques. The GLM weights were created by mixing existing 2.05 and 3.05 bpw (bits per weight) EXL3 tensors with a custom layer mix and a small tuning stage. The developers noted that while their 100 GB mix improved Kullback–Leibler (KLD) divergence compared to standard 2.05 bpw packs, the smaller 85 GB version decodes approximately 10% faster.
Technical Specifications and Availability
The Kyojin engine is currently available via GitHub, with model weights hosted on Hugging Face under the yamz-labs repository. The developers have also released separate “Uncensored” variants of the models, which can be toggled via a single switch during the loading process.
While the current benchmarks focus on the Ryzen AI Max+ 395, the developers noted that performance metrics for other GPUs and task-suite scores at a 128K context window have not yet been measured. The conversion pipeline used to create these weights remains private.
The project includes a quickstart guide for users with Strix Halo hardware, allowing for the deployment of an OpenAI-style API through a standard build script.