AI Models Approach Human Baseline on SimpleBench

Recent performance data from SimpleBench indicates that frontier artificial intelligence models are beginning to reach the human baseline in specific reasoning tasks. SimpleBench is a benchmark designed to evaluate capabilities that large language models (LLMs) have historically found difficult, including commonsense reasoning, spatial and temporal reasoning, social judgment, and the ability to avoid shallow pattern-matching or misleading assumptions.

According to unverified reports circulating on social media platforms like Reddit, current model scores are being measured as an average accuracy across five runs. For context, a random guessing strategy yields a score of 16.7%, a level near which models were performing approximately two years ago. The human baseline used for comparison was established by nine native-English, non-specialist participants with at least high-school-level mathematics proficiency.

Data shared by users suggests a competitive landscape among major AI developers. While specific measurements for OpenAI’s “Astra” are reportedly pending, Google’s Gemini models have demonstrated consistent capability on this specific benchmark. Some observers noted that Gemini’s performance may be linked to its proficiency in following instructions.

However, the utility of such benchmarks remains a subject of debate among industry observers. Some commentators argue that high scores on specialized benchmarks may not always reflect general model capabilities, suggesting that developers have significant financial incentives to “max-out” specific tests to drive investment and engagement. Critics point to instances where custom harnesses were developed specifically to improve scores on benchmarks like ARC-AGI, raising questions about whether high scores indicate genuine intelligence or optimized performance for a specific test.

Limitations in “Soft Skills” and Practical Application

Despite improvements in reasoning benchmarks, some users report that progress in practical, “soft” skills has been inconsistent. For example, it has been noted that while certain models, such as MoonshotAI’s Kimi K3, show improved coding and agent performance, they may experience regressions in text analysis compared to previous iterations like K2.6.

Additional concerns have been raised regarding the following:

  • Writing Quality: Some users report that latest-generation models struggle to navigate from a hypothesis to a conclusion, often producing convoluted sentences.
  • Adversarial Robustness: The ability of models to resist adversarial input and maintain a specific role remains a challenge, often requiring multiple models to verify inputs and outputs.

While benchmark scores are rising, some analysts suggest that the most reliable metric for AI value remains commercial adoption and the willingness of companies to pay for the technology, citing the rising annual recurring revenue (ARR) of major players like OpenAI and Anthropic as indicators of real-world utility.