Independent Developer Proposes Training of 9.4B Parameter Model
An independent machine learning developer has announced plans to train a new dense large language model (LLM) featuring approximately 9.4 billion parameters. According to discussions on the Reddit community LocalLlama, the developer intends to release the model, tentatively named “Budget,” as an open-source project.
The proposed architecture utilizes a 1/2/3 Engram table, Moonshot’s AttnRes modeling, and RoPE/NoPE layering at a 3:1 ratio. The developer noted that the model’s initial state uses the Llama 3 series tokenizer and LM Head, with training data derived from logit-level extraction from a Llama 3 teaching model.
The developer stated that the training code is optimized for single-card execution on an RTX 6000 Pro series GPU. While the developer claims the model should theoretically outperform existing 9B parameter models if provided with sufficient training, the project is being conducted outside of a formal laboratory setting, utilizing a combination of a personal RTX 4090 and rented hardware.
Focus on Data Augmentation and “Thinking” Capabilities
The developer indicated that the focus of the instruction tuning (IT) phase will be to imbue the model with “thinking” capabilities, following emerging open standards.
Community feedback on Reddit has raised technical questions regarding the implementation of AttnRes. One user, Marcuss2, suggested that Gated Residuals, as seen in Qwen3.8 Flash Next, may offer superior performance.
There is also significant interest from users regarding the model’s utility in document processing. One contributor, ttkciar, expressed interest in a 9B model optimized for document cleaning and augmentation. The user noted that while larger models like Gemma-4-12B-it perform these tasks effectively, they are “fat” in terms of parameters and K/V caches. A leaner 9B model would theoretically allow for larger batch sizes and more efficient parallel processing of documents within limited VRAM.
The proposed training methodology involves complex data augmentation, combining cleaning and augmentation into a single task to maximize inference-time economy. This process aims to replicate methods used by IFM (formerly LLM360) to produce high-competence models for data extraction and analysis.
Market Context and Infrastructure Requirements
The project remains in a speculative stage, as the developer has not yet secured a datacenter partner to host the full pre-training and instruction tuning. The developer has stated a willingness to contribute the code and data to the public domain.
The announcement comes amid ongoing discussions regarding the sustainability of AI investment. While some commenters on Reddit have speculated about an “AI bubble” or a potential slowdown in the sector, others argue that the utility of the technology distinguishes it from previous speculative bubbles, such as NFTs. In a related development noted in daily discussions for September 14, 2026, Alphabet Inc. was reportedly viewed by some observers as having been “marked safe” from such market volatility.