Technical Milestone in Browser-Based AI Inference

A solo developer has reported a significant performance milestone for Mentria.ai, a browser-based inference engine built from scratch using WebGPU and WGSL. According to reports on Reddit, the engine has achieved a decoding speed of 25–30 tokens per second using a 27-billion parameter model on a laptop equipped with an RTX 3060 GPU (6 GB VRAM) via the Chrome browser.

The engine utilizes Bonsai-27B, a natively 1-bit model released by Prism ML. The developer has repackaged the model to operate at approximately 1.14 bits per parameter, allowing the 27B model to occupy roughly 3.8 GB of GPU memory. This configuration enables the model to run locally within a web page without requiring software installation or server-side processing, ensuring that data remains on the user’s machine.

Engineering Optimizations and Performance Metrics

The reported performance increase from 15 to 30 tokens per second is attributed to specific kernel optimizations. The developer noted that the decoding process is memory-bound, requiring 804 GPU dispatches per word. To address this, a specialized 1-bit matrix-by-vector kernel was implemented.

Initial testing showed that the kernel faced hardware-specific bottlenecks on Ampere architecture due to memory bank conflicts. By implementing a padding slot per row to spread addresses across the 32 memory banks, the developer reported that the kernel’s share of a word dropped from 26.5 ms to approximately 21 ms.

Key performance benchmarks reported for the RTX 3060 (Windows 11, Chrome 152 on D3D12) include:

  • Decoding Speed: 25–30 tokens per second in the chat UI.
  • Prompt Processing: Approximately 25 seconds for a 1,489-token prompt.
  • Context Window: 3,072 tokens on 6 GB cards; up to 8,192 tokens on 16 GB Mac systems.
  • Loading Time: Under 10 seconds from the browser cache after an initial 3.8 GB download.

The engine also supports smaller model tiers (0.8B, 2B, and 4B) for mobile and low-end devices, as well as LoRA adapter hot-swapping and a vision tower for image input. While these technical details are documented in the project’s GitHub repository and engine facts page, these figures are provided by the developer and have not been independently verified by third-party benchmarks.

Broader AI Industry Sentiment

The technical developments in local inference occur amid a broader, more contentious discourse regarding artificial intelligence. On September 10, 2026, discussions on social platforms reflected a spectrum of concerns ranging from safety to market stability.

In political spheres, the UK Prime Minister has confirmed discussions regarding AI risk, following warnings from researchers regarding potential catastrophic outcomes. Simultaneously, the industry faces internal scrutiny, with reports of employees resigning from AI-focused roles due to safety concerns. Within retail investor circles, some participants have expressed skepticism regarding the long-term viability of the current AI market, with some characterizing the sector as a “bubble.”