🔍 Read the full analysis: FrontierCode: What Developers Can Learn From AI Pull Request Reviews on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository tasks judged for correctness, scope and review quality. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts, while cost per task varied sharply across models.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that scores whether a proposed repository change would meet maintainers’ review standards. Claude Opus 5.5 ranked first with a 52.2% score, but it cleared all blocking checks in only 57.7% of attempts, underscoring that the leading result does not establish reliable autonomous pull requests.
Artificial Analysis added the benchmark to its evaluations on October 12, 2026, using an open-source agent harness called Stirrup. Each model had up to 250 turns per attempt and three attempts per task. The evaluation covered the benchmark’s 100-task Main set, drawn from real software repositories.
GPT-6.1 Sol ranked second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Xiaomi’s MiMo-V2.6-Pro scored 45.4%, the highest result among the open-weight models listed. The benchmark’s overall score is distinct from its pass rate: an attempt that fails any blocking check scores zero, while attempts that clear those checks receive scores from 70% to 100%, depending on quality criteria.
Task costs also differed substantially. Artificial Analysis reported $10.15 per task for the leading Opus configuration and $0.81 for GPT-6.1 Sol. The evaluation found nine of the 100 tasks were not solved by any model on any attempt. These are results from this benchmark and setup, not a guarantee of how a model will perform on a particular team’s codebase.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode focuses on a practical question for engineering teams: would maintainers accept the change? Its results show why a green test suite alone may not be a sufficient measure of AI-generated code. A patch can pass tests yet still touch too many files, add weak tests or violate repository expectations.
For developers, the figures suggest that AI tools can assist with implementation but still require human review, especially where changes affect reliability or maintenance. The price spread adds a procurement consideration: teams may need to weigh the incremental benchmark score against the cost of running a model. The reported difference between Sol and Opus was 3.1 percentage points at a much higher listed cost for Opus; that comparison does not establish which is more economical for every workload.
The benchmark also cautions against selecting coding tools solely from general-purpose intelligence rankings. In the source’s comparison, Claude Sonnet 5.5 ranked second among the listed models on the Intelligence Index but fifth on FrontierCode; DeepSeek V4.1 Flash ranked higher on FrontierCode than its Intelligence Index position might suggest. That is evidence of different rankings in this comparison, not proof that one benchmark predicts every development task.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Pull Requests
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July, adding rules against searching online for task solutions. More than 20 maintainers created tasks based on 36 repositories, including Celery, Budibase, uppy and Mattermost. The full benchmark contains 150 tasks; Artificial Analysis used the 100-task Main set. The tasks are private, according to the source, to reduce the chance that models encountered them during training.
Scoring covers six areas: hidden tests, command checks, tests that must fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM-based code review against maintainers’ criteria. The review judge in this run was GPT-6 Luna (medium). Artificial Analysis said attempts flagged by Cognition’s trajectory scanner for looking up solutions scored zero; oversized diffs, out-of-scope edits and unhelpful tests also affected scores.
That design shifts attention from whether a change can be made to whether it is appropriately scoped and useful to maintainers. Celery maintainer Tomer Nosrati summarized the distinction in Cognition’s announcement: “Where others grade like a CI, FrontierCode grades like a tech lead.”
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The results do not show how every model performs across all 150 FrontierCode tasks, since Artificial Analysis ran the 100-task Main set. They also reflect a particular harness, attempt limit and review judge; changing those conditions could affect model rankings. The source reports the benchmark’s scoring and cost figures, but does not establish that they predict performance on individual companies’ repositories.
Several models had no result listed as of October 13, 2026, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence leaves the comparison incomplete. The source also does not report independent replication of these leaderboard results or explain how the listed task costs would translate to different usage patterns.
network attached storage for developers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Missing Models and Further Runs
The next useful development will be results for the models missing from the October 13 comparison, followed by additional evaluations that can show whether rankings hold across different tasks and setups. Artificial Analysis’s published figures provide a point-in-time comparison; they do not establish a permanent ordering as models and evaluation methods change.
Developers considering AI-generated pull requests can use the benchmark as one reference, while checking tools against their own review standards, repositories and costs. For now, the clearest result is that even the top-ranked model did not clear all blocking checks in more than half of its attempts under this evaluation.
AI-powered pull request review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates repository changes against correctness, regression, test quality, scope and code-review criteria. A model must pass all blocking checks to receive a nonzero score.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the Artificial Analysis run with a 52.2% FrontierCode score. Its reported pass rate, meaning attempts clearing every blocking check, was 57.7%.
Does a 52.2% score mean the model passes tests 52.2% of the time?
No. The benchmark score and pass rate are separate measures. The source reports a 57.7% pass rate for Opus; its 52.2% score reflects the benchmark’s scoring system, which also grades quality after blocking checks are passed.
How much did the tested models cost per task?
Reported costs ranged from $0.09 to $12 per task across the models in the source. The leading Opus configuration was listed at $10.15 per task, while GPT-6.1 Sol was listed at $0.81; actual costs may depend on usage and setup.
Can teams use the results to replace human code review?
The benchmark does not establish that. The leading model failed to clear blocking checks in 42.3% of attempts in this run, so the results support treating human review as still necessary rather than assuming an AI-generated change is ready to merge.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
