Technical Specifications of the Engram Model Update
A developer has released technical details regarding an update to the “Engram” model, a small-scale large language model (LLM) currently in training. According to reports on the r/LocalLLaMA subreddit, the model architecture has been locked at a total of 2.6 billion parameters. This includes a 2.2 billion parameter trained backbone, with the embedding and LM head layers—each approximately 205 million parameters—remaining frozen. The developer utilized Singular Value Decomposition (SVD) to compress these layers from OLMo 3, reducing them from approximately 5k to 2k to optimize training efficiency.
The architecture incorporates a 4.3 billion parameter “Engram table” and utilizes Attention Based Residuals (AttnRes). This mechanism allows subsequent blocks to access the Engram table via the residual stream, determining the necessity and volume of data injection. The model features 41 layers in total, consisting of 10 blocks followed by a final global layer, with a 3:1 Sliding Window Attention (SWA) ratio.
Performance Observations and Training Progress
As of the latest update, the model has surpassed the 100 million token training mark. The developer reported that the model utilizes a Knowledge Distillation (KD) format, employing 32 soft targets from a teacher model trained on the Wikipedia corpus rather than standard one-hot encoding.
Preliminary inspections of the model after 100 million tokens suggest that the Engram table facilitates “offloading” of data. The developer noted that as the model encounters repetitive data, such as early Wikipedia entries focused on philosophy, it relies more heavily on the Engram table rather than internal computational space. In comparative testing, the Engram-enabled model provided more detailed completions for historical and philosophical prompts compared to a version where the Engram table was zeroed out.
Independent experimentation by other users on the same platform has yielded similar, though unverified, results. One user reported that adding a 120 million entry Engram table to a MiniCPM-2B model resulted in improved accuracy for medical-specific queries. In their reported benchmarks, the Engram-enhanced model showed a slight improvement in Top-1 accuracy (48.77% vs 48.68%) and a reduction in cross-entropy loss (2.6580 vs 2.6624) compared to the base model.
Economic Implications of Declining Intelligence Costs
The technical advancements in small-scale, efficient models like Engram coincide with broader discussions regarding the declining cost of intelligence. In online forums, observers have noted that the accessibility of AI-driven instruction may disrupt traditional education business models, as low-cost AI subscriptions may replace high-priced specialized courses.
The ongoing development of these models highlights a shift in how compute is allocated, moving from massive, general-purpose training toward specialized, table-based data offloading. While the Engram project currently relies on self-funding, the developer has expressed interest in securing corporate sponsorship to complete the training process.