New Deterministic Benchmark Targets LLM Context Management

A new evaluation framework, LabyrinthBench, has been released to measure how large language models (LLMs) manage long-term context during multi-step agentic tasks. According to documentation from the project’s GitHub repository, the benchmark is designed to be “judge-free,” meaning it relies on deterministic scoring rather than using another LLM to grade the performance of the model under test.

The benchmark focuses on “context recall under interference,” specifically testing whether a model can retrieve information from earlier in a conversation as the dialogue grows longer. This addresses a common technical hurdle in AI agent workflows, where “context compaction” or hitting token limits often causes models to lose track of critical instructions or data provided in previous turns.

Methodology and Test Structures

LabyrinthBench utilizes a series of “maps” to test different levels of cognitive load. While an initial version included navigation through a maze, the developer has transitioned to a “20-gate corridor” model to focus specifically on memory and reasoning. In this setup, each “gate” requires the model to solve a problem that relies on the answer from a previous gate, forcing the model to maintain an accurate internal record of the conversation.

The testing includes three distinct families of challenges:

  • Simple Gates: Basic arithmetic and logic evaluations.
  • Chained Gates: Problems where each subsequent answer depends on a specific earlier value.
  • Dynamic Variable Gates: Challenges where a variable is initialized, changed mid-way through the sequence, and then queried again, requiring the model to recompute rather than simply repeating a previous answer.

Initial Experimental Results

The developer released preliminary data from an experiment involving 13 local models. The study compared two conditions: maintaining the full chat history versus “wiping” the context every turn and re-injecting only the model’s previous gate answers.

The results, shared via a Reddit announcement, showed varying impacts of context wiping:

  • Performance Gains: Seven of the nine models capable of showing improvement saw higher median gate depths when context was wiped. Notably, the deepseek-r1:70b model moved from a median of 1 gate to 20 gates when using the wiping method. The qwen3:14b model reached a median of 20/20 gates under these conditions.
  • Performance Declines: Two 70B-class models, llama3.3:70b and llama4:scout, saw performance decreases when context was wiped. In the case of llama3.3:70b, traces indicated the model repeatedly submitted incorrect answers because the wiped context lacked a record of previous failures.

The data also noted that while context wiping can improve depth, it can significantly increase costs; one model required 21 times more output tokens to achieve its depth compared to the control group.

Implementation and Accessibility

LabyrinthBench is available as an open-source project on GitHub. It is designed to run locally via Docker Compose, allowing users to test models on their own hardware without requiring API keys for local models. For cloud-based models, the framework currently requires a proxy.

The project includes a leaderboard, though the developer notes that rankings are based on a conservative statistical bound (a 95% bootstrap lower confidence bound on median depth) to prevent single lucky runs from skewing the results. While the benchmark provides a new method for objective measurement, it remains a self-published tool and has not yet undergone independent academic peer review.