New Benchmarks Suggest AI Superiority in Experimental Research Taste
Recent data emerging from the research community suggests that artificial intelligence models are beginning to outperform human researchers in evaluating “experimental research taste.” According to findings associated with the TasteEval project, AI models have demonstrated a capacity to identify high-quality research directions more effectively than human counterparts.
While the results indicate a shift in how research quality is assessed, some observers on social platforms like Reddit have characterized these findings as a “baby version” of true research taste, suggesting that the benchmark may only capture a subset of the complexities involved in scientific intuition.
Discrepancies in Automation and Leadership
Despite the performance of AI in specific evaluative tasks, a distinction remains between AI-assisted research and fully automated research. Unverified claims circulating in online discussions suggest that while approximately 26% of research may currently be led by AI, the percentage of research that is fully automated remains at 0%.
This distinction is critical as the industry debates the timeline for Artificial Super Intelligence (ASI). Rayan Krishnan, Co-Founder of Vals AI, has suggested that Anthropic may reach Full Research Super Intelligence (RSI) by August 2027. However, these projections remain speculative and lack independent, peer-reviewed confirmation.
Technical Limitations and Resource Competition
The ability of AI to outperform humans in research taste appears to be tied to specific computational advantages. Analysts note that these models excel at incremental engineering and the tuning of parameters to optimize for specific metrics. This capability allows them to navigate structured environments more efficiently than humans.
However, limitations persist in open-ended problems and highly uncertain environments where human intuition has traditionally been required. Furthermore, the physical requirements of scaling these models have sparked debate regarding resource allocation. As AI models require increasing amounts of physical space for data centers and power plants, as well as significant mineral resources, questions are being raised regarding the competition for space and materials between human infrastructure and the expansion of AI hardware.
Verification and Data Integrity
The scientific community remains cautious regarding the release of new benchmarks. Some discussions have highlighted the importance of rigorous data verification, noting that unreleased models or incomplete datasets can lead to premature conclusions. As the field moves toward more automated research processes, the industry continues to monitor whether these models can transition from optimizing specific metrics to navigating the broader, more complex landscape of scientific discovery.