Coding agent benchmarks: what actually won?

If you're choosing a coding agent, a benchmark comparing two complete setups can be useful. You care about which one gets the work done. But if you want to know which model is better, you need to look at what else changed.

That distinction came up in a Reddit discussion about parallelizing scientific code. The underlying evaluation project runs models through different coding tools, including Codex CLI and Claude Code. Its methodology explicitly records those differences. The comparison is useful, but the model name alone does not explain the result.

The harness is the software surrounding the model. It lets the agent inspect files, run commands, edit code, and continue after a tool returns a result. One harness might retry a failed edit automatically; another might stop and wait for help. Those choices affect how quickly the work gets done and whether it gets done at all.

For a platform team evaluating agents, this changes how to read the leaderboard. A winning setup is a reasonable candidate to test. It is not yet an explanation of why that setup won, or evidence that the same model will win inside your team's tooling.

The harness changes the work

Imagine a controlled experiment with the same model and the same repository in two different harnesses. In this hypothetical task, the agent has to fix a function that fails a unit test.

The first harness returns the failing assertion and the surrounding stack trace. The model can see the mismatch, inspect the relevant function, and try a targeted edit. The second returns a long command transcript with the useful failure buried in the middle. If that output is truncated before the model sees it, the next step may be another search or another test run just to recover the missing information.

The model has not changed, but the work available to it has. That extra investigation consumes time and tokens, and it may lead the agent down a less useful path. A similar difference can come from how the harness chooses files, summarizes previous work, or decides when to stop.

Testing policy matters too. Running a full suite after every edit can catch mistakes early, but it also adds waiting time. Running no tests makes generation look fast while leaving someone else to discover whether the patch works. Neither policy can be understood from the model label on a chart.

This is why a benchmark result belongs to the setup that produced it: the model, the harness, their configuration, and the conditions of the run. The useful question is how closely that setup resembles the one you would actually use.

Two comparisons answer different questions

A team choosing between complete coding products should usually start by testing those products as it would use them. Give each the same tasks and starting repository, allow a reasonable configuration, and compare the outcomes. A product's better file handling or test integration is part of its value.

Replacing the model inside an existing agent is a different decision. Here, keeping the harness steady helps you see what the model change contributes. The tools, instructions, retry allowance, and tests should stay as consistent as the APIs permit. Provider differences may still prevent a perfectly identical setup, so those differences belong beside the result.

The existing comparison diagram illustrates this distinction. Comparing two models in two different harnesses measures the complete setups. Comparing models within the same harness gives you a better chance of separating the model's contribution from the surrounding software.

There is no need to turn every purchasing decision into a research project. If your team intends to use the complete product, the complete-product result may be exactly what you need. The mistake is carrying that finding into a different setting and assuming the model will bring the entire advantage with it.

Fast code still has to work

Once the comparison is clear, the next question is what counts as finishing the task. Producing a patch and producing a useful result can take different amounts of time.

Consider another hypothetical example. Agent A finishes a parallelized calculation in nine minutes, while Agent B takes fourteen. Both patches compile, and both pass the small test cases. On a larger input, however, Agent A's changed calculation order produces a numerical difference beyond the test's accepted tolerance. Agent B's output remains within that tolerance.

The extra five minutes bought a result that passed the required check. If A needs another attempt, its initial finish time no longer describes how long it took to get working code. If it never passes, it should remain a failed attempt in the comparison.

The scientific-code project's methodology makes a useful distinction here: generation time is recorded separately from later validation and performance testing. Generated programs must pass output comparison before they qualify for performance benchmarking. That is worth understanding before interpreting its time charts; the time shown for producing code is not the same measurement as the time spent evaluating it.

For a local trial, keep the acceptance tests outside the agent's authority to redefine. The agent can run tests while it works, but a final clean run should check the result against the task you assigned. Our guide to live-inference agent tests develops this point for serving changes, where a patch must also produce the intended behavior under a workload.

A trial your team can use

Start with a small set of representative tasks from your own work. A test repair, a dependency update, and a performance change may reveal different strengths, so keep their results visible rather than immediately averaging them into one score.

Each candidate should begin from the same clean repository state. Decide beforehand what a passing result requires and how much time or repair work is allowed. Repeat tasks where practical, because one successful run can hide inconsistency.

A compact run record is enough to make the comparison inspectable:

Cost labels need care as well. The evaluation project's API costs are estimates based on recorded token volumes and pricing profiles, rather than observed bills. For your own trial, distinguish an estimated API cost from an actual charge, and keep engineer time visible separately.

Use the external benchmark to narrow the candidates, then let representative work determine the choice. If one complete setup reliably finishes your tasks with less waiting and less intervention, that is a useful result. You can choose it without pretending the leaderboard has isolated every reason it performed better.