A new Alibaba-led benchmark tested 10 AI coding agents on 273 real user requests: the best one fully finished only 26.4% of them. Where that number comes from, why benchmarks promised 99%, and how to build with agents without paying for your product twice.
- Published
- Reading
- 12 min
- Author
- Yaroslav Tsarenko

The agent met 70% of the requirements. Sounds nearly done, right? Now the second number from the same study: tasks fully completed — 26.4%. Almost three out of four had at least one detail that didn't work.
Those numbers come from Co-Design Bench, a benchmark that researchers from Alibaba and several universities published on October 26, 2026. The tech press picked it up in early November, and it explains rather well why your product has been "almost working" for three weeks now.
This article is for founders and product teams building with Cursor, Claude Code, Codex, Lovable or Bolt — or hiring a contractor who works with agents. We'll go through what was actually measured, why older benchmarks promised 99%, where agents break most often, and how to set up the work so you don't pay for the same product twice.
What Co-Design Bench actually measured
Most coding benchmarks are simple: there's a task, there are unit tests, the agent passes or it doesn't. Co-Design Bench is built differently, and that's exactly why it's worth reading.
- Tasks come from real life. The authors collected 37,463 unique requests from 26,314 real sessions where people asked an agent to change an app, and curated 273 tasks out of them.
- Two tracks. 100 tasks are localized edits (Multimodal Editing, often with an image or document in the request) and 173 are coordinated changes across several files of a repository (Repository Editing).
- 2,331 hidden acceptance criteria written by experts. The agent never sees them — just like your contractor never sees what you "obviously meant".
- The running app is what gets graded. It's built in a sandbox, opened in a browser with Playwright, and an AI judge walks through every criterion. A second judge challenges every "pass" and looks for mock-ups and hard-coded results.
- 10 models in the same harness: the same tools (bash, read, search, edit files), the same conditions.
The headline metric isn't the average share of requirements met — it's full-pass: the share of tasks where everything works. Just like a real product: nobody pays for a checkout that's "mostly" done.
70% of requirements, 26% of tasks: why "almost" isn't done
The winner was gpt-6-astra. It met 69.9% of the criteria per task on average but fully completed only 26.4% of tasks. On localized edits: 41%. On multi-file repository changes: 17.9%. So when a task touches several parts of an app, the best model on the market gets it all the way over the line roughly once in every five or six attempts.
Sound familiar? The agent built the form, the validation, the styles, even an animation. Then it turns out the Save button hides under the keyboard on mobile, a server error never shows up, and old records without the new field break the list. Each of those "small things" isn't 1% of the work. Sometimes it's half the time, because first someone has to figure out what's wrong in code nobody wrote by hand.
Here's the full table from the paper:
| Model | Requirements met, avg | Tasks fully done | Localized edits | Multi-file changes |
|---|---|---|---|---|
| gpt-6-astra | 69.9% | 26.4% | 41.0% | 17.9% |
| opus-5 | 66.1% | 18.3% | 30.0% | 11.6% |
| qwen3.8-max | 65.3% | 16.5% | 31.0% | 8.1% |
| kimi-k3 | 62.9% | 15.8% | 26.0% | 9.8% |
| qwen3.8-27b | 64.4% | 15.0% | 23.0% | 10.4% |
| gpt-5.6-sol | 61.1% | 14.3% | 20.0% | 11.0% |
| glm-5.3-flash | 57.3% | 10.6% | 18.0% | 6.4% |
| opus-4-8 | 58.1% | 10.3% | 16.0% | 6.9% |
| glm-5.2 | 56.7% | 9.5% | 16.0% | 5.8% |
| grok-4.6 | 51.1% | 6.6% | 6.0% | 6.9% |
Notice the pattern? The spread in "average requirements met" is up to 19 points. The spread in fully finished tasks is fourfold. The last details are exactly where models pull apart.
An important caveat: these tasks are hard on purpose
Honesty over headlines. The authors didn't pick random requests. They picked ones where users rephrased the request several times or rolled back the agent's changes — tasks that had already given agents trouble. They name two more limits themselves: the agent couldn't ask a clarifying question mid-task, and the judges are models too, which can be wrong.
So "agents only finish a quarter of all tasks" is the wrong takeaway. The right one: on hard changes to a real product, even the best agent almost always leaves something unfinished. And hard changes are precisely what you pay engineers for. An agent can recolor a button without any benchmark telling you so.
Why benchmarks promised 99%
If you read model launch posts, you've seen numbers like 80% on SWE-bench or close to 100% on editing benchmarks. The Co-Design Bench authors say plainly that older editing benchmarks are saturated: models score above 99% and the leaderboard barely separates them. On the new test the gap exceeds 35 points.
They're not alone.
- OpenAI stopped reporting SWE-bench Verified on February 23, 2026. It audited 138 problems its model consistently failed and found serious flaws in the tests or problem statements of 59.4% of them — plus signs that models had seen the reference solutions during training.
- METR asked 4 maintainers of scikit-learn, Sphinx and pytest in March 2026 to review 296 AI-written pull requests that passed SWE-bench's automated tests. Roughly half would not be merged into main. After adjusting for the fact that maintainers only accepted 68% of the original human patches, the gap versus automated grading is 24 points. Top reasons: code quality, breaking other code, core functionality not working.
To be fair, METR stresses this isn't a capability ceiling — agents had one attempt and no feedback, unlike a developer in code review. But that's how a lot of people actually use agents, isn't it? One attempt, "looks fine", merged.
Where agents break most often
Co-Design Bench offers a few practical hints that map directly onto how you write tasks.
Several kinds of change in one request
The authors split changes into four types: layout, design, interaction and content. All 10 models did worse on repository tasks mixing three or four types than on those with one or two. "Redo the pricing page: new grid, new colors, a monthly/yearly toggle and new copy" is exactly that kind of request.
Changes across several files
The best model finishes a localized edit 41% of the time and a repository change 17.9% of the time. The more places that must change in sync, the likelier one gets missed. In a typical product, almost any new feature touches the database, the API, the UI and access rules at once.
More tokens doesn't mean better
My favorite result. On localized edits GLM-5.2 spent a median of 69 turns and about 7.29M tokens per task and reached 57.2% of requirements. gpt-6-astra: 32 turns, 1.24M tokens, 75.4%. An agent that "thinks" for ages and rewrites everything isn't necessarily doing better work. Sometimes it's just going in circles, and your API bill grows along with it.
The harness matters as much as the model
The same qwen3.8-27b met 74.9% of requirements on repository changes in one agent harness (opencode) and 56.0% in another (qwen-code). Almost 19 points, same model. The authors recommend always reporting the model together with the harness. For you, the takeaway is simpler: a model leaderboard guarantees nothing in your project. What matters is everything around the model: tools, instructions, tests.
Usual
- Everything in one promptlayout, logic, styling and copy at once
- "Make it right"the criteria live only in your head
- Eyeball check"looks done", merged
Works
- One kind of changelayout first, logic separately
- Written criteria5–10 checkable points before starting
- Tests and reviewCI, types, a person reads the diff
What real-world usage studies show
A benchmark is a lab. What happens when real developers use agents?
SWE-chat (accepted at COLM 2026) analyzed nearly 18,000 real developer sessions with coding agents. A few numbers:
- in 41% of sessions the agent wrote practically all the committed code — vibe coding in its purest form;
- only 59% of agent-written code survives into commits — the rest gets thrown away or rewritten;
- in about half of all turns users corrected the agent, reported a failure or interrupted it;
- agent-written code introduces more security vulnerabilities than human-written code.
And the preprint (Im)Paired Programming adds an uncomfortable detail about the humans. In a study with 54 students, those working with an agent finished the task faster but understood their own code less and couldn't extend it on their own. They still preferred the agent, because it's quick and easy.
Spoiler: that's the most expensive part. When nobody on the team understands how the code works, every next change costs more. I covered this in 10 signs your vibe-coded MVP needs a developer — the "AI going in circles" sign is exactly this.
What this means for your product
Short version: agents are a strong tool for the draft, not the finish. That's not pessimism; it's accounting.
Where agents already save money:
- a prototype to validate an idea or a demo for investors;
- standard screens: lists, cards, forms, filters;
- routine changes with clear criteria: rename a field, add a column, move a component;
- tests, migrations and scripts that a person then reviews.
Where "almost done" costs the most:
- access control — every gap means someone else's data in the wrong hands;
- money — payments, refunds, duplicate webhooks;
- data — migrations you can't roll back;
- features that touch several parts of the system at once.
The cost of a mistake here isn't "redo a button" — it's a data leak, double charges or lost orders. Why, for example, AI API keys are hunted especially hard right now and what to do if one ends up in your code: see my breakdown of the LiteLLM and Railway key leaks.
How to work with agents without paying twice
Everything below follows directly from the data above. No magic, just discipline.
- One task, one kind of change. Split "redo the page" into separate tasks: layout, logic, copy. Co-Design Bench says plainly that mixed changes fail more often.
- Acceptance criteria before starting, in writing. 5–10 points, each one checkable: "after payment the order status is paid", "user B gets a 404 on user A's order". The agents in the benchmark failed on hidden criteria. Don't hide yours.
- Automated checks in CI. Tests on critical flows, type checks, a linter. The agent misses details; the machine catches them before your customer does. How fast type checking is in TypeScript 7 now and how to wire it into CI: see my migration guide.
- Every change as a pull request. A person reads the diff, not just the result in the browser. Especially access control, money, database queries and error handling.
- Fix the harness. One agent, one set of project instructions, the same rules. Nineteen points from the harness alone is not a rounding error.
- Watch the spend. If the agent has been rewriting the same thing for three hours, it isn't "trying hard" — it's stuck. Stop it, split the task or hand it to a person.
- Someone on the team must understand the code. Not all of it, but the critical parts — definitely. Otherwise every next change gets more expensive.
A 10-minute checklist for your project
| Question | Yes | No — what to do |
|---|---|---|
| Are there automated tests for sign-up, payment and data access? | start with 5–10 tests on these flows | |
| Do tests and type checks run on every change? | set up CI; it's half a day of work | |
| Do agent tasks come with written criteria? | template: what must work + how to check it | |
| Does someone review every change before merge? | at least review access, money and data | |
| Can you roll back a release in 5 minutes? | set up rollback to the previous deployment | |
| Can someone on the team explain how payments and permissions work? | time for an audit or a developer |
Three or more "no" answers mean the product has outgrown "the agent builds, we look at it".
What to ask a contractor who works with agents
In 2026 almost every developer uses agents, and that's fine. The question isn't "do you use AI" — it's "how do you check what it wrote". Five questions whose answers tell you a lot before the project starts:
- "How do you decide a task is done?" Good answer: acceptance criteria, automated tests, review. Bad answer: "I check that it all works."
- "Which parts of the product do you never hand to an agent without review?" A good contractor names access control, payments and database migrations immediately.
- "What happens if the agent breaks something that used to work?" Listen for "tests in CI" and "release rollback", not "I'll notice and fix it."
- "Who will explain how the code works six months from now?" Docs, a clear structure, and critical parts a person can explain without the agent.
- "How much do AI requests cost during development, and who pays for them?" A detail — until the agent gets stuck on one task for a few hours.
If most answers sound like "well, AI can do everything now", you'll get a product that looks finished. About 70% finished, as we now know.
When an agent needs a person next to it
An agent doesn't replace an engineer, but it amplifies one who knows where to look. The typical picture when founders reach out to me: the product was built fast, it works in the demo, but with the first real users come duplicate records, strange statuses and fear of changing anything.
The work is almost always the same: a written risk audit, close the most dangerous gaps first (data access, keys, money), add tests and CI, and then keep building calmly — with agents, too, but on rails. The AllShip case shows what that looks like in practice: three fragile codebases with no consistent authorization layer became one app with access checks on every one of its 109 entry points.
What this costs and what drives the price, I covered separately in MVP development cost in 2026. And if your product already works and you need to know what's worth saving, there's a dedicated service: rescue for AI-built apps.
The short version
- The best AI agent in the new benchmark meets about 70% of requirements but fully completes only 26.4% of real tasks — 17.9% on multi-file changes.
- Older benchmarks are saturated: OpenAI dropped SWE-bench Verified, and METR showed maintainers would reject roughly half of "passing" patches.
- Agents struggle more with mixed changes, and results depend on the harness as much as on the model.
- An agent is a great tool for the draft. The finish — access, money, data and edge cases — still needs criteria, tests and a person who understands the code.
Agents will get better, and fast. But the gap between "almost" and "done" isn't going away; it's just getting easier to miss. If you'd like someone to look at your product with fresh eyes and tell you what's actually finished, get in touch.
Was this useful?
– likes · – views
Questions people ask
Can an AI coding agent build a working app on its own?
A prototype or a single feature, often yes. But in the Co-Design Bench study the best of 10 agents fully completed only 26.4% of real change requests to an existing app, even though it met about 70% of the requirements on average. The agent usually does most of the work; the last details still need a person.
Why do benchmarks show 80–99% when real results are lower?
Older benchmarks are saturated and partly leaked into training data. In February 2026 OpenAI stopped reporting SWE-bench Verified after finding flaws in 59.4% of the problems it audited, and METR found that maintainers would not merge roughly half of the SWE-bench patches that pass the automated tests.
What is the best AI model for coding in November 2026?
In Co-Design Bench, gpt-6-astra came first (26.4% of tasks fully done, 69.9% of requirements met), followed by opus-5 and qwen3.8-max. But the authors show the same model can move almost 19 points depending on the agent harness, so pick based on your own tasks rather than a leaderboard.
How do I check code written by an AI agent?
Write acceptance criteria before the agent starts, run tests and type checks in CI, review every change as a pull request, and look separately at access control, payments and data handling. For a product that already exists, the fastest route is a short audit with a written list of risks.