New Benchmarks Test AI Decision Models in Real-Time Environments
A new wave of specialized AI “decision models” is undergoing rigorous testing as developers move beyond standard large language model (LLM) metrics toward real-time interaction capabilities. Recent unverified performance data shared by independent researchers on Reddit highlights a significant divergence in how these models handle high-speed, latency-sensitive tasks.
Pac-Man Performance and Latency Constraints
In a recent benchmark reported on Reddit, six popular decision models—including OpenAI’s “kev” (versions 1.13 and 4B), Cloudflare’s “Clef” and “Clef Flash,” “GPT-6 Luna,” and “Laya”—were tested by having them play the classic game Pac-Man. The test aimed to evaluate decision-making consistency in a real-time environment where response times are measured in milliseconds.
According to the results published by the researcher, “jev 1.13” achieved the highest average score of 2,750, followed closely by “GPT-6 Luna” at 2,568. “Clef Flash” and “Clef” recorded average scores of 2,538 and 2,476, respectively. The “Laya” model recorded the lowest average score of 639, though it demonstrated the lowest average latency at 104 ms.
The benchmark utilized a 100-run sample size for each model, calculating the mean score with a 95% margin of error. The open-source repository for this test allows for the inclusion of locally hosted or fine-tuned models. While the results show varying levels of success, commenters noted that latency remains a critical factor in real-time interaction, with some models struggling to maintain accuracy when response times increase.
Hardware-Specific Speed and Accuracy Testing
Separate testing conducted on an NVIDIA RTX 4090 GPU focused on the raw processing speed and accuracy of decision models when performing repetitive, word-by-word classification tasks. The test required models to read 9,534 words from Wikipedia articles and identify specific terms.
The results, also shared via Reddit, indicate a significant gap in processing speed between models. “Laya” emerged as the fastest, processing words at a rate of 3.9 ms per word, totaling 7,980 words in 32 seconds with a 97.4% accuracy rate. “d1 3B” followed with a speed of 6.0 ms per word.
In contrast, “Clef-Flash 9B” recorded a significantly higher latency of 24.4 ms per word, and “Lev 4B” recorded 51.0 ms per word. While “Lev 4B” achieved the highest accuracy in the test at 98.9%, it was 13 times slower than “Laya.”
Industry Implications and Limitations
The emergence of these models—including Liquid’s d1, Cloudflare’s Clef-Flash, and Interfaze’s Lev—suggests a growing niche for models optimized for specific, low-latency decision-making rather than general-purpose conversation.
However, independent observers noted that current performance in specialized tasks remains inconsistent. In the Wikipedia classification test, even the highest-performing models struggled with “wrong picks,” leading some researchers to suggest that while these models show promise for tasks like “relevance filtering,” they may require further fine-tuning or different architectural approaches, such as embeddings, to be viable for production-level applications.