← Blog

Practice & mistakes

70 percent done, 26 percent right: The AI coding gap

Markierte Pull-Request-Ausdrucke und ein roter Stift liegen neben der Tastatur.

Monday starts with a bug report. The new filter looks right, but returns the wrong results for certain entries. A coding agent prepared the change. Now a person has to find what is missing and check whether other features are affected. A benchmark using real development tasks exposes the gap between “looks finished” and “works properly”. The best model tested met around 70 percent of a task’s requirements on average. It produced a fully correct result far less often. How can you tell whether an agent is still taking work off your team’s hands?

70 percent sounds better than it is

The figure sounds reassuring at first: GPT-6 Astra, the best of ten models tested, met around 70 percent of a task’s requirements on average. But the result was fully correct for only 26.4 percent of tasks. For a development team, that is an important distinction. A half-finished suggestion can help you get started, but the job is not done.

The Co-Design Bench tests whether an agent can change an existing web app in response to a user request. The tasks come from real coding sessions. In some cases, users had to rephrase their requests several times or undo changes. That is closer to everyday work, where a requirement is unclear and a fix raises new questions, than to a neat test with one right answer.

The researchers also included inputs such as images, audio recordings, videos and documents. That reflects something many teams know from experience: a ticket does not always contain the whole request. But what does “done” mean if the interface looks polished while an important workflow still fails?

A polished screen is not an acceptance test

A green tick on the task list proves little if nobody has tried the result. In the benchmark, an AI judge tests the finished app in a browser. A second AI judge checks positive assessments again, looking, among other things, for mock-ups or hard-coded results. The agent gets credit only for features that actually work.

There is a clear practical lesson for your team: write acceptance criteria that can be observed. If a search function is being changed, “improve search” is not enough. The question is whether a user can enter the expected term and get a relevant result. The implementation may look technically sound and still miss the point.

An AI-based test is still a test you need to understand and control. That is why the benchmark uses a second check. In your organisation, this could mean an automated test alongside human review of cases where an error would be costly. Which cases should your reviewers try to catch first?

Complex requests are the stress test

One small change is rarely the whole job. A form may need to look different on a phone, a button may need to trigger a new action, and the explanatory text may need an update too. The benchmark finds that when tasks combine several types of change, every model tested performs worse.

This is not just a problem for large development departments. A small or medium-sized business can run into the same tangle in its customer portal, internal tools or ordering process. You ask for a change to the layout and discover during review that a workflow, a piece of content or an existing feature is affected too. The agent has to handle several dependencies, while the request may be no more than a short line in the ticketing system.

GitHub recommends writing issues with context and acceptance criteria, then splitting them up where needed. Smaller, focused pull requests are easier to review than one large change covering the interface, logic and text at once. If the work genuinely belongs together, keep it together. But how do you stop splitting it up from creating more admin?

What a fair in-house test needs

An agent can do well in a general benchmark and still struggle with your conventions, tests or data structures. Assess candidates on your own code and against your own acceptance criteria before giving them production work. The results may look less impressive, but they will be much closer to the decision you actually need to make.

Use tasks from day-to-day work. Choose cases where the expected result is known and someone can judge whether it works fully. Record what guidance the agent needed, how many corrections were required and whether the change passed review. A simple suggestion can be useful even if it does not complete the task on its own. You just need to know what kind of help you are paying for.

GitHub recommends recording team conventions and validation commands in repository instructions. That gives the agent and the people on your team the same guidance, including how code should be checked. For an initial test, you could ask agents to redo a small selection of completed tasks, then compare the results blind. Which measure will tell you whether that creates an advantage?

  • Choose tasks that come up regularly in your team and have an outcome you can check.
  • Write down the required features and quality standards before the test.
  • Check the finished result with tests and review, not just by looking at the code output.
  • Record human rework and errors alongside time spent and AI costs.

Count merged changes, not just code

The number of lines of code generated is tempting to track, but it does not tell you whether your team can deliver faster. A more useful measure is the cost per merged pull request: the total cost of AI-assisted development divided by the number of reviewed changes that are actually merged.

That measure needs company. Track rework and errors too, so an apparently cheaper change does not come at the expense of quality. A quick draft can shift work from writing to reviewing, correcting and debugging. You may only notice in day-to-day work when the pull request stays open and a developer runs through the same workflow for the third time.

GitHub cites a Faros AI study in which pull requests took 441 percent longer to review as teams moved from low to high AI adoption. That does not prove AI always makes reviews longer. It does show why an adoption rate or token count alone tells you little about the value you are getting. What would count as a win for your team if review takes longer?

What this means for your business

Next Monday, an agent could take on a small, clearly defined bug. The issue sets out the context and acceptance criteria. The pull request contains a focused change that passes automated checks. A developer then checks whether it works in your particular workflow. That turns an AI demo into a work trial you can assess.

Start with a task whose success is clear-cut. Record how long the work takes, including review, how much rework is needed and whether the change is merged. Compare that with the current process. If an agent only partly solves a task, that is not automatically a failure. You need a reliable answer to whether the remaining human work has gone down or simply shifted elsewhere.

A small team may need no more than a short checklist. A medium-sized business should also track team standards and recurring errors. At a large organisation, you may need a consistent way to compare and accept work across different repositories and owners. The right test will not be the same size everywhere. But the question is the same: what counts as a delivered result in your organisation?

Frequently asked questions

How do I test a coding agent in my business?

Choose tasks from your everyday work where the expected result is clear. Set acceptance criteria in advance and let the agent work with your code and repository rules. Then check that the feature works, review the tests and human rework required, and see whether the change passes review. A general benchmark alone cannot show how well an agent fits your team.

What does the 26.4 percent figure mean for coding agents?

In the Co-Design Bench, GPT-6 Astra was fully correct on 26.4 percent of tasks. On average, it met around 70 percent of the requirements. The figures measure different things: an agent can complete much of a task without producing a result that works fully or meets every requirement.

How do I measure the value of AI in software development?

One possible measure is the cost per merged pull request. Divide the total cost of AI-assisted development by the number of reviewed changes that are accepted and merged. Track rework and errors as well. This gives you a better idea of whether AI is reducing effort, rather than simply producing more code or suggestions.

Why should coding tasks be split into smaller pull requests?

Smaller, focused pull requests are easier to review than a large change affecting many features at once. That makes errors and deviations easier to spot. GitHub recommends splitting tasks where needed and recording requirements, context and acceptance criteria. Changes that genuinely belong together should still be treated as one connected task.

Can a coding agent replace human code reviews?

The evidence available does not show that coding agents can replace human reviews. In the Co-Design Bench, the finished app is tested and positive AI assessments are checked again. In your business, you should assess results against your tests and acceptance criteria, and plan for human review where errors would have consequences.

Interested?

Want to know what this means for your company?

Leave your details. We will get back to you personally, no sales pressure and no newsletter sign-up.

Sources

  1. AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test · the-decoder.com, 2026-11-08
  2. Beyond code generation: A developer’s guide to agentic engineering · github.blog, 2026-11-06
  3. Stop counting tokens, start counting what ships · technologyreview.com, 2026-11-02

This article was researched and written with AI assistance and automatically checked against the sources listed before publication. More on our AI transparency

Free guide · PDF

The AI Compass for small and mid-sized businesses

9 jobs your team can hand over by Monday.

Get it free

Klarframe AI Readiness Check

How ready is your company for AI?

About ten minutes, instant result with your stage, a 90-day plan, suitable solutions and notes on the EU AI Act.

Start the check

More articles