New Architecture Proposed to Decouple Knowledge Updates in Large Language Models
Researchers have introduced EngramEdit, a new method designed to update factual knowledge within Large Language Models (LLMs) without requiring the retraining of the entire model backbone. According to the research paper, the method utilizes conditional memory architectures—specifically n-gram embeddings—to separate factual storage from general-purpose computation.
Technical Framework and Methodology
The EngramEdit framework builds upon conditional memory architectures, such as DeepSeek Engram, which use input n-grams to look up learned embeddings. This approach aims to expand the capacity of LLMs with minimal additional computation.
A primary challenge in knowledge editing is that different linguistic expressions of the same fact may activate different n-gram embeddings. Furthermore, updating shared embeddings can inadvertently alter the model’s predictions regarding unrelated facts. To address this, EngramEdit computes target memory representations that ensure the model predicts an updated fact across multiple expressions. The method jointly updates shared n-gram embeddings to match these targets while applying a penalty to frequently reused embeddings to preserve existing, unrelated knowledge.
Performance Benchmarks
Experimental results cited in the research indicate that EngramEdit achieves near-perfect editing success. The study reports that revised knowledge remains usable across unseen expressions and within multi-hop reasoning tasks. Notably, the method demonstrated an accuracy nearly three times higher than the strongest baseline when utilizing chain-of-thought (CoT) prompting. The researchers state that general capabilities and unrelated knowledge remain largely preserved even as multiple factual updates are accumulated.
Implementation and Community Discussion
The project has been made available via GitHub and HuggingFace, with the full technical details hosted on arXiv.
In discussions on the social platform Reddit, some users have noted that the process functions similarly to fine-tuning, as it requires running the model to edit its own n-gram weights rather than simply loading random weights. Some commentators suggested that this could realize the theory of “swappable n-gram tables,” allowing developers to update a model’s knowledge base to current information without the necessity of training an entirely new model.
However, some technical skepticism remains regarding the scalability of these edits. Unverified comments on Reddit raised questions regarding “sequential edits” versus “batched edits,” noting that previous methods like ROME and MEMIT have shown degradation after thousands of individual updates. There were also inquiries regarding whether edits remain stable when faced with paraphrases that share no n-grams with the original training expressions.