Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence

📊 Full opportunity report: Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

All six major benchmarks launched in 2023-2024 to evaluate AI research and engineering skills have now saturated or are approaching saturation within a few months, according to Thorsten Meyer’s analysis. This pattern indicates that AI capability is advancing at an accelerated pace, with significant implications for industry, policy, and research timelines.

Thorsten Meyer reports that six benchmarks designed to challenge AI systems in various aspects of research and engineering have all either been saturated or are tracking toward saturation within a short timeframe. These benchmarks include SWE-Bench, METR Time Horizons, CORE-Bench, MLE-Bench, PostTrainBench, and CPU Speedup, each measuring different facets of AI development.

For example, SWE-Bench, which assesses real-world software engineering tasks, improved from 2% to 93.9% in 30 months, reaching a point of high performance. Similarly, METR Time Horizons, measuring task duration completion, expanded from 30 seconds to 12 hours over four years, representing a significant increase. The CORE-Bench, focused on research reproduction, was declared solved by its authors after reaching 95.5% in just 15 months. These rapid progressions are consistent across all six benchmarks, indicating a pattern of accelerated saturation.

This pattern, described as a ‘saturation cascade,’ suggests that AI research capabilities are approaching or surpassing human-level performance across multiple domains on a timeline of months rather than years, challenging previous assumptions about the pace of AI development.

Implications of Rapid Benchmark Saturation for AI Development

The saturation of all six key benchmarks within a short period demonstrates that AI systems are rapidly approaching or exceeding human-level performance in core research and engineering tasks. This acceleration has implications for AI deployment, industry innovation, and policy regulation, as capabilities once thought to take years to develop are now materializing within months.

Stakeholders in AI development, including companies, governments, and researchers, should consider the evolving timelines, safety considerations, and ethical frameworks. The pattern suggests that AI progress may soon reach a point where it can autonomously perform complex research tasks, potentially influencing the pace of AI innovation and its societal impact.

Background on Benchmark Development and Progress

Since 2022, researchers have introduced a series of challenging benchmarks aimed at measuring the core capabilities of AI systems across research, engineering, and deployment tasks. These benchmarks, such as SWE-Bench and METR Time Horizons, were explicitly designed to be difficult and to push AI systems toward high performance.

Over the past three years, progress has been notable, with each benchmark showing significant improvements. Notably, the SWE-Bench, which tests real-world software engineering skills, improved from 2% to nearly saturation at 93.9% in just 30 months. Similarly, METR Time Horizons expanded from 30 seconds to 12 hours over four years. The CORE-Bench, measuring research reproduction, was declared solved in late 2025 after reaching 95.5%.

Thorsten Meyer’s analysis indicates that the simultaneous saturation across these diverse benchmarks signals a pattern of rapid capability advancement, contradicting earlier projections of slower progress.

“All six benchmarks launched in 2023-2024 have saturated or are nearing saturation within months, indicating that AI development is progressing at an accelerated rate.”

— Thorsten Meyer

Uncertainties Around Long-Term AI Capability Trajectories

While the saturation of these benchmarks indicates rapid progress, it remains uncertain how this translates to broader AI applications and real-world deployment. Some experts question whether benchmark saturation equates to general intelligence or only narrow skill mastery. Additionally, the long-term sustainability of this rapid pace, potential plateaus, or regressions are still uncertain, as are the implications for safety and regulation.

Next Steps in Monitoring AI Benchmark Saturation

Researchers and industry stakeholders will continue to track these benchmarks to confirm whether saturation persists and to observe if new benchmarks emerge. Further analysis will focus on how these capabilities translate into practical AI systems and what regulatory or safety measures need to evolve in response. Expect updates on the development of more comprehensive benchmarks to measure general intelligence and safety in AI systems.

Key Questions

What does benchmark saturation mean for AI development?

Benchmark saturation indicates that AI systems have achieved or exceeded human-level performance on specific tasks, suggesting rapid advancement in AI capabilities within those domains.

Are these benchmarks representative of general AI progress?

Not necessarily. These benchmarks measure specific skills and tasks; saturation in these areas does not automatically imply AI has achieved general intelligence or can perform all human tasks.

What are the risks of such rapid progress?

Accelerated capability development raises concerns about safety, control, and ethical implications, especially if AI systems become autonomous in research and deployment without adequate oversight.

Will new benchmarks be developed to measure future capabilities?

Yes, researchers are likely to create more comprehensive and complex benchmarks to evaluate broader aspects of AI intelligence and safety as current ones saturate.

How soon might AI systems autonomously conduct research?

Based on current progress, some AI systems are already approaching the ability to perform research tasks autonomously; full autonomy could emerge within the next few years, depending on continued advancements and safety measures.

Source: ThorstenMeyerAI.com