FrontierCode: Does Your AI’s Pull Request Meet Maintainer Expectations?
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: FrontierCode: Does Your AI’s Pull Request Meet Maintainer Expectations? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Artificial Analysis published results for 23 models on FrontierCode, a benchmark that evaluates whether AI-generated code changes meet repository maintainers’ criteria. Claude Opus 5.5 ranked first, but cleared all blocking checks in 57.7% of attempts; the benchmark’s results do not establish how models perform across every real-world repository.

Artificial Analysis published results on October 12 for 23 AI models tested on FrontierCode, a benchmark that judges coding agents by whether their changes meet maintainer-style review criteria, not just whether tests pass. Claude Opus 5.5 ranked first with a 52.2% overall score, but cleared every blocking check in 57.7% of attempts, leaving substantial room for human review.

FrontierCode’s distinction is its scoring standard: an attempt that fails any blocking check scores zero, even if other tests pass. The benchmark also assesses patch quality, including whether tests catch regressions, whether changes stay within scope, and how a language-model review rates the code against maintainers’ criteria. Attempts flagged for looking up solutions online also receive zero.

Among the 23 models evaluated, GPT-6.1 Sol ranked second at 49.1%, followed by Claude Fable 5.1 at 49.0%. Artificial Analysis reported that nine models from six labs scored between 43% and 52%. On the separate pass-rate measure, GPT-6.1 Sol cleared all blocking checks in 54.7% of attempts, while Claude Fable 5.1 did so in 54.3%.

Artificial Analysis ran models through the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. Its evaluation used the benchmark’s 100-task Main set; the full FrontierCode benchmark contains 150 tasks. The organization reported that nine of the 100 tasks were not solved by any model on any attempt.

At a glance
reportWhen: Results published October 12, 2026; sta…
The developmentArtificial Analysis added FrontierCode results for 23 coding models, ranking Claude Opus 5.5 first while showing that its attempts cleared all blocking checks 57.7% of the time.
Crypto market snapshot
Fear & Greed Index
57/100 — Greed
Bitcoin BTC$83,269▲ 0.8%
Ethereum ETH$2,506▼ 0.8%
Tether USDT$0.9992▲ 0.0%
BNB BNB$742.55▼ 0.3%
XRP XRP$1.37▼ 0.7%
USDC USDC$0.9998▲ 0.0%
Solana SOL$106.81▼ 1.9%
TRON TRX$0.3273▼ 1.0%
Live data · CoinGecko · alternative.me (24h change)
Thorsten Meyer AI
FrontierCode 1.1

Infographic 01 · Would the maintainer merge it?

The best coding model clears the bar 58% of the time

FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.

52.2%Top score: Claude Opus 5.5 (max with fallback). GPT-6.1 Sol follows at 49.1%, Claude Fable 5.1 at 49.0%.
42%of the leader’s attempts fail a blocking check. Opus 5.5 clears every blocker in 57.7% of attempts, the highest pass rate.
9 / 100tasks no model solved on any attempt. No task was solved by every model every time.

FrontierCode score, 23 models

Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.

  1. Claude Opus 5.5 (max with fallback)Anthropic52.258% pass
  2. GPT-6.1 Sol (max)OpenAI49.155% pass
  3. Claude Fable 5.1 (max with fallback)Anthropic49.054% pass
  4. GPT-6 Astra (max)OpenAI47.052% pass
  5. Claude Sonnet 5.5 (max with fallback)Anthropic45.851% pass
  6. MiMo-V2.6-ProXiaomi45.451% pass
  7. DeepSeek V4.1 Flash (max)DeepSeek44.049% pass
  8. Kimi K3 (max)Kimi43.649% pass
  9. Qwen3.8 Max (0902)Alibaba43.248% pass
  10. GPT-6 Luna (max)OpenAI41.947% pass
  11. Claude Haiku 5.5 (max)Anthropic41.847% pass
  12. GLM-5.3-FlashZ AI41.046% pass
  13. Muse Spark 1.3 (max)Meta40.646% pass
  14. GLM-5.3 (max)Z AI40.145% pass
  15. Step 5 PreviewStepFun38.443% pass
  16. Gemini 3.8 Flash (high)Google36.441% pass
  17. Qwen3.8 27B (xhigh)Alibaba34.238% pass
  18. Grok 4.7 (xhigh)SpaceXAI33.838% pass
  19. K2 Horizon 375B A23BInstitute of Foundation Models32.136% pass
  20. MiniMax-M3MiniMax23.026% pass
  21. Inkling (xhigh)Thinking Machines15.818% pass
  22. Muse Glimmer (high)Meta13.916% pass
  23. Nemotron 3 UltraNVIDIA10.5

Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.

Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI

Why Passing Tests Is Not Enough

The results highlight a gap between producing code that passes tests and producing a change a maintainer would accept. FrontierCode checks for issues that ordinary test outcomes may miss, including unnecessary edits, weak tests, regressions, and departures from task scope. That makes it relevant to teams considering AI agents for repository work: a successful test run alone does not show that a patch is ready to merge.

The leading result is also a measure of limits, not a guarantee of reliability. Claude Opus 5.5 cleared all blocking checks in 57.7% of attempts, according to Artificial Analysis. The figure leaves many attempts that did not satisfy every blocking requirement. It does not mean that 57.7% of all AI-generated pull requests in everyday use would be accepted; the reported result applies to this benchmark and its testing setup.

Cost adds another practical consideration. Artificial Analysis listed Claude Opus 5.5 at $10.15 per task and GPT-6.1 Sol at $0.81, with respective overall scores of 52.2% and 49.1%. Those figures suggest a substantial price difference for a relatively small score gap in this evaluation. They are benchmark costs, not a full accounting of deployment, review, or engineering expenses.

Products Worth Considering

Apple MacBook Air 13-inch Laptop with M5 Chip, 16GB Memory, 512GB SSD, 12MP Center Stage Camera, Touch ID, Wi-Fi 7, Midnight

Apple MacBook Air 13-inch Laptop with M5 Chip, 16GB Memory, 512GB SSD, 12MP Center Stage Camera, Touch ID, Wi-Fi 7, Midnight

DJI Mic 2 Wireless Microphone System (2 Transmitters, Receiver, Charging Case)

DJI Mic 2 Wireless Microphone System (2 Transmitters, Receiver, Charging Case)

As an affiliate, we earn on qualifying purchases.

How FrontierCode Scores Patches

FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules intended to prevent models from finding solutions online. According to the source report, more than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy, and Mattermost, spending more than 40 hours on each task. The tasks are private so models cannot have encountered them in training, the report says.

The evaluation uses six kinds of checks: hidden tests for correctness; command checks for regressions and cleanliness; reverse tests to establish that an agent’s tests fail on the original code; adaptive tests for complex behavior; limits on files and diff size; and an LLM-based code review against maintainer criteria. Some checks are blocking and others affect the quality score. Passing all blocking checks produces a score between 70% and 100%, depending on quality checks, according to the supplied report.

Artificial Analysis’s results are specific to its run configuration: three attempts per task, a 250-turn cap, and GPT-6 Luna (medium) as the review judge. The report also notes that several models had no result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full, and Mistral Large 4 Preview. Those missing results mean the published ranking does not cover every model named in the report.

“Where others grade like a CI, FrontierCode grades like a tech lead.”

— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement

Limits of the Published Results

The results describe one benchmark and a defined agent setup; they do not establish how often the same models would produce acceptable changes across all repositories, programming languages, or team review practices. The supplied report does not provide enough detail here to independently assess variation across repeated runs beyond the stated three attempts per task.

Several named models had no FrontierCode score as of October 13, and the reason for each missing result is not specified. The 100-task Main set is also only part of the benchmark’s 150 tasks. How rankings might change on the full set, with different agent configurations, or under other review judges remains unknown.

Results to Watch Next

The next useful development will be additional FrontierCode results for models absent from the October 13 leaderboard, along with any updates to the benchmark’s task set or evaluation method. Those results could change comparisons among the tested systems, but no release timetable is specified in the supplied material.

For readers comparing these scores, the published pass rate and overall score answer different questions: the pass rate records attempts that cleared every blocking check, while the overall score also reflects quality scoring. Both should be read alongside the task count, run limits, and stated cost per task rather than treated as a prediction of acceptance in a particular software project.

Key Questions

What does FrontierCode measure?

It evaluates whether AI coding agents produce repository changes that satisfy maintainer-oriented checks, including correctness, regression testing, scope limits, and an LLM-based code review. A failed blocking check gives the attempt a score of zero.

Which model ranked first in the reported results?

Claude Opus 5.5 (max with fallback) ranked first with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts, according to Artificial Analysis.

Does the leading result mean AI pull requests are ready to merge without review?

No. The benchmark result is not evidence that AI-generated pull requests can routinely be merged without human review. Even the top model failed to clear every blocking check in a substantial share of its attempts, and the evaluation covers a defined set of tasks and settings.

How many tasks were tested?

Artificial Analysis ran the 100-task Main set. The full FrontierCode benchmark contains 150 tasks, and the supplied report says nine of the 100 tested tasks were not solved by any model on any attempt.

What remains unknown about the rankings?

The results do not show how models would rank on the full benchmark or across all real-world repositories. Several models had no reported result as of October 13, 2026, and the source does not specify when those results may appear.

Source: ThorstenMeyerAI.com

Nothing in this article is financial or investment advice. Cryptocurrency and precious-metal investments carry significant risk — do your own research and consider a licensed advisor.
You May Also Like

One Video In, a Whole Publishing Kit Out — Without the Cloud

Discover how to turn a single video into a full publishing package without uploading to the cloud. Fast, local, and control-rich workflows for creators.

Why AI Capital Could Be Getting Harder To Win

Reports say OpenAI cut its fundraising target to $20 billion, while S&P warned about Meta’s AI spending. The round remains unclosed.

The Emerging AI Power Metric: Agents Per Gigawatt Explained

Exploring how the emerging measure of agents per gigawatt redefines economic and national AI power, linking energy capacity to autonomous cognition.

The Machinery Of AI: An Inside Look At Twelve Key Systems

A detailed look into twelve core systems powering AI chatbots, revealing how they work, why they matter, and what remains uncertain about their operation.