Advancements in Local Inference and Model Performance

Developments in large language model (LLM) inference efficiency and specialized model architectures are surfacing across open-source communities. Recent reports from user-driven benchmarks and developer releases highlight significant variations in processing speeds and reasoning capabilities.

STRATA Optimization and Local Inference Speeds

New performance claims regarding “STRATA” optimization have emerged via user reports on Reddit. According to a user-submitted benchmark, the STRATA framework—described as a specialized fork of llama.cpp tuned for a specific model—demonstrated a significant increase in tokens per second (t/s) compared to standard configurations.

The user reported achieving 9 t/s using Unsloth 3-bit Qwen 3.8 Flash Next on a standard rig. However, after implementing STRATA on CachyOS, the reported speed increased to peaks of 48 t/s, with an average of 30 t/s. While these figures suggest a substantial improvement in local inference, it should be noted that these results are unverified by independent third-party testing and stem from individual hardware configurations.

The reports also highlighted the constraints of consumer-grade hardware. On an 8GB GPU, the user reported that a 32k context size functioned effectively, while a 42k context size pushed the hardware to within 130MB of a VRAM-related crash.

DeepSeek Releases DeepInfer Framework

In a move to enhance large-scale deployment, DeepSeek has released “DeepInfer,” an open-source high-performance inference framework designed specifically for DeepSeek models. The release consists of two primary repositories:

  • DeepInfer: A framework intended for high-performance, large-scale inference.
  • DeepSeek-Recipe: A set of libraries written in Rust with Python bindings, designed for protocol conversion and model output parsing. This component aims to connect inference backends to various API formats, including Chat Completions and Messages.

Agnes-AI Announces Agnes-3.0-Qwen

Agnes AI has released Agnes-3.0-Qwen, a multimodal reasoning model utilizing a hybrid attention Mixture-of-Experts (MoE) architecture. The model contains approximately 180 billion total parameters, with 10 active experts per token.

According to technical specifications provided by Agnes AI, the model features a 1,048,576-token context window and supports text, image, and video inputs. The model is designed with selectable reasoning effort levels: xhigh (default), medium, and low.

In preliminary benchmarks, the model reportedly achieved high scores in agentic performance, specifically on the $\tau^3$-Banking dataset, and performed competitively on Terminal-Bench 2.1. Additionally, the model claims to outperform GPT-5.6 Sol (Max) and Claude Opus 4.8 (Max) on the IFBench instruction-following benchmark. The model is released under the Qwen Community License 1.0.