26.4%. That’s the share of actual person app-edit duties that the strongest coding agent absolutely cleared on Alibaba’s new Co-Design Bench, in keeping with a paper from Alibaba Token Hub and college co-authors. The identical run met 69.9% of every activity’s hidden acceptance standards on common. Most jobs have been principally completed and nonetheless unfinished.
The Co-Design Bench paper (arXiv, 26 October 2026; undertaking website at codesignbench.github.io) builds 273 duties from 37,463 deduplicated queries throughout 26,314 actual coding periods. Brokers should revise a runnable internet app from a hard and fast person request, generally with picture, audio, video or doc attachments. Specialists wrote 2,331 hidden acceptance standards. A main AI choose grades the completed app from supply code and browser proof; a second verifier challenges each Cross or Partial verdict and appears for stubs, hardcoding and unreachable options.
What the headline numbers imply
OpenAI’s GPT-6 Astra led ten mannequin endpoints below a shared Claude Code harness. Throughout all 273 duties it scored 0.699 imply strict completion — roughly seven in ten standards handed per activity — and a 26.4% full-pass charge. Learn these collectively: on 73.6% of duties a minimum of one criterion nonetheless failed. Common requirement protection isn’t the identical as delivery a whole change.
The hole is sharper by observe. On 100 Multimodal Modifying duties, Astra absolutely handed 41.0% and averaged 0.754 strict completion. On 173 Repository Modifying duties — coordinated adjustments throughout parts, routes, state and knowledge circulation — it absolutely handed 17.9% with 0.667 strict completion. Repository edits that spanned three or 4 change varieties scored decrease for each endpoint than these with one or two varieties. Pooled strict completion fell from 0.654 on narrower edits to 0.591 on broader ones.
That sample matches how folks really transient coding instruments. A request that solely tweaks one CSS property isn’t the identical job as one which rewires navigation, state and a dependent abstract in a single go. The benchmark’s repository observe is constructed so these compound asks are frequent: 166 of 173 repository duties require a number of edit varieties.
Why older assessments look too straightforward
The authors constructed the set as a result of older modifying benchmarks are saturating as soon as fashions can loop, run and repair their very own code. On 150 WebCompass-edit duties, holding the rubric and choose mounted, Qwen3.7-Plus rose from 73.6% imply graded characteristic completion in a single shot to 95.1% with agentic execution; Claude Opus 4.8 reached 99.4% agentic on that very same cut up. Scores that top cease separating merchandise.
Co-Design Bench as an alternative favoured session requests that already confirmed friction — reformulations, follow-ups or rollbacks — and grades solely options that work within the working app. Multimodal duties can ask an agent to tug a worth from a receipt or a video clip and hold each dependent complete and examine constant. Repository duties can couple clip-count limits, timeline accuracy and restoration after a cut up. A believable native patch can nonetheless go away a linked view flawed.
Want AI to save you hours a week? Our free playbook shows you how.Get the free guide →
Who led, and who didn’t
Behind Astra, Anthropic’s Claude Opus 5 and Alibaba’s Qwen3.8-Max adopted on mixed full-pass charge (18.3% and 16.5%). xAI’s Grok 4.6 completed final at 6.6% full passes. Rankings shifted between tracks: Qwen3.8-27B rose to second on repository edits even whereas rating decrease on multimodal ones. The paper doesn’t deal with that as a common mannequin benefit — solely as proof {that a} single mixed board can disguise setting-specific strengths.
Burning extra tokens didn’t repair the sample. On multimodal work, Astra reached 0.754 strict completion with 32 median turns, whereas GLM-5.2 used 69 turns and a number of other instances the token site visitors for a decrease rating. Additional compute helped some runs grind longer with out closing the unfinished standards that block a full go.
The harness strikes the scoreboard
When the staff held two solvers mounted and swapped the agent harness — Codex, OpenCode or Qwen Code — scores moved by as much as 16.2 share factors on repository edits. Which of the 2 fashions led additionally relied on the harness. On multimodal edits the harness ordering even reversed between solvers. The paper’s sensible line is blunt: report mannequin and harness collectively. A leaderboard that names solely the mannequin tells a part of the story.
For an organization selecting a coding agent, meaning testing the pair you’ll really run — mannequin plus tooling — on edits that appear like your backlog, not trusting a chart that lists the mannequin alone.
What the check doesn’t declare
Limits are said within the paper. Every activity is a hard and fast request; the agent can’t ask follow-up questions or get mid-task corrections. AI judges can nonetheless err, although two frontend consultants checked rubric scores on 200 activity–mannequin outputs. The end result measures autonomous completion below a strict rubric, not interactive work with an individual within the loop. It is usually not a measured charge of how typically people clear up bizarre day-to-day coding. The sincere studying is narrower: on onerous, real-session revisions, even the very best endpoint left one thing unfinished more often than not.
That caveat matches a theme DailyAI has tracked elsewhere. METR has argued that published speed claims can outrun the evidence people can check, and OpenAI’s personal safety brokers lately found 948 Linux privilege-escalation flaws in a paid research run — a reminder that agent scores and actual supply are completely different measurements.
Why it issues
Distributors nonetheless promote coding brokers with near-perfect scores on older assessments. On this real-session set, the very best endpoint left one thing unfinished on about three in 4 duties. Anybody shopping for or deploying an agent ought to ask for full-pass charges on real looking edits, and for the harness these numbers got here from, not solely common criterion protection.

