New Research Proposes Application of Recursive Self-Improvement to Model Pre-training
Researchers have released a paper detailing a new approach to Recursive Self-Improvement (RSI) applied directly to the pre-training phase of large language models. While previous RSI research has focused on post-training—where an existing base model undergoes iterative trials to improve verifiable scores—this new methodology targets the entire production system during the initial creation of a foundation model.
According to the research paper titled “No Second Run: Does RSI Stop at Frontier Pre-training?”, applying RSI to pre-training presents significant logistical challenges. Unlike post-training, which can be repeated with relatively low GPU costs, frontier pre-training involves hundreds of thousands of GPU-hours over several months and cannot be easily repeated. Furthermore, the researchers note that the objective of pre-training is not a single score, but rather the “post-training headroom” available to the model later.
The 801-30B-A3B-Base Model
To overcome the inability to run an end-to-end loop over a single, unrepeatable pre-training run, the researchers utilized a method of decomposing the process into smaller, manageable loops. Each loop is designed to repair specific steps within the broader production pipeline, which includes data pipelines and training infrastructure.
The result of this process is the 801-30B-A3B-Base, a 30B-A3B Mixture-of-Experts (MoE) model. The model was trained from scratch on 11.8 trillion tokens over a three-month period by a newly formed team. While the researchers have released the base model and lightweight Supervised Fine-Tuning (SFT) data via GitHub, the full model weights are listed as “coming soon.”
Performance Benchmarks
The researchers report that the 801-30B-A3B-Base model demonstrates competitive performance against existing industry models. According to the paper, the base model outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B.
In tests using lightweight SFT (6 billion tokens without Reinforcement Learning), the model reportedly achieved the highest average score across four agentic benchmarks compared to fully post-trained models, including the Qwen3.6-35B-A3B. The researchers suggest that by applying RSI to the pre-training stage, they are improving the fundamental foundation upon which all subsequent training stages are built.
Community Reception and Technical Skepticism
Following the release of the findings on platforms such as Reddit, the technical community has offered mixed perspectives. Some users expressed skepticism regarding the practical utility of the results, suggesting that high performance on specific benchmarks may indicate “overfitting” rather than a fundamental improvement in model intelligence for real-world applications. Other commenters questioned whether the RSI process described was a genuine advancement or a method of “skinning the benchmark” to achieve higher scores.
As of the time of reporting, the claims regarding the model’s superiority in agentic benchmarks remain based on the authors’ reported data and have not been independently verified by third-party benchmarks.
