Independent research · pre-registered · MIT
Where does agent-harness engineering actually pay off? Same models, same tasks, the harness stripped arm by arm, graded by held-out verification tests the agent never sees.
In production the harness visibly earns its keep. Across a fleet running coding agents for up to four hours per task, telemetry over a recent window shows 28.8% of dev-class runs – 106 of 368 – hitting at least one deterministic guardrail block: dirty-tree stops, force-push prevention, done-without-PR gates. Each one is a divergence that prose instructions had been letting through.
Measured under controlled conditions at the scale most coding benchmarks operate – single-session, minutes-long, well-scoped tasks against a well-tested repository – the same layers did nothing at all. Both facts are true, which is the interesting part: harness ROI is a function of horizon, not a constant. The pilots bound that curve from below and the production telemetry bounds it from above. The study is an attempt to find the knee.
Two target repositories, both pinned and both private during every graded run: a 20K-line TypeScript monorepo with 732 tests, and a 31K-line pipeline. Tasks are synthetic; briefs are fixed; only the harness varies. The arms are cumulative.
| Arm | Adds |
|---|---|
| A0 | Bare prompt. The agent self-reports done. |
| A1 | Deterministic done-gate: a Stop hook that blocks the agent from finishing until the full suite is green (max 3 blocks). |
| FULL | Adds a schema-forced result contract with one salvage retry, plus an adversarial reviewer agent with one revision round. |
| Layer | What it checks |
|---|---|
| Visible suite | The target repository's own 732 tests. |
| Tamper check | Any modified or newly added test file fails the run. |
| Held-out verification | Assertions the agent never sees, kept outside the working tree and injected only at scoring time – calibrated green on pristine and red under every planted bug before any graded run. |
A run succeeds only if the visible suite and the held-out assertions both pass and no test file was touched. A run that passes the visible suite while failing the holdout is recorded as a silent failure – the agent finished, and nothing it could see said otherwise.
The design, arms, metrics and cost caps were committed to the repository before the corresponding graded runs, and every mid-flight correction is an append-only event in the journal next to the original record rather than an edit to it. Pilot 7 added a calibration gate: a full reference implementation of all eight issues, replayed commit by commit, requiring every held-out test to be red on the state before its issue and green after.
Pilots 1–2 minutes scale – the null
Five single-line mutations with the failing tests named in the brief, then eight harder tasks in four classes: discovery, compound, behavior-report and feature. 54 of 54 graded runs succeeded, across every arm, both models and all five classes – including tasks whose planted bug left all 732 visible tests green, so that only a held-out suite could grade the fix. The done-gate never fired, because the agents already ran the suite unprompted, and the reviewer never caught a defect, because there were none to catch. Haiku with no harness at all matched Sonnet with the full harness at roughly 40% of the cost. The harness’s only measurable effect at this tier was about 2× the spend.
Pilot 3 hour scale – the onset
Three cross-cutting feature builds spanning multiple packages: a new worker mode wired through five layers, a refactor behind a pluggable queue interface, a webhook flow. The frontier model still did not need the harness. The small model did, decisively: haiku went 1/3 bare and 3/3 with the full harness, matching sonnet’s outcomes at roughly a third of the cost, at n=1 per cell. Its two failures are the production-relevant kind – a correct abstraction shipped with five repository tests left broken mid-migration, and a 200 shipped where the spec said 202, asserted by the agent’s own added tests.
Pilot 4 the planning ablation
Give that same failing executor a written implementation plan and the spec-misreading failures disappear completely: planned haiku went 17/18 bare across three planner tiers, including haiku planning for itself, against 1/3 unplanned. The one failure that survived was the follow-through relapse – walking away mid-migration – and only the enforcement loop closed it reliably. Plan for the what, harness for the whether.
Pilot 5 the sharp edge of planning
The replication: 72 runs over six new tasks across two repositories, n=3. Exactly one configuration went perfect – plan plus full harness, 18/18 – and bare-with-plan went 14/18, below bare-without-plan’s 16/18. The autopsy is the finding rather than the proportion: a plan substitutes for exploration, so when its consistency sweep is incomplete the executor stops at the checklist instead of discovering the gap. No comparison here reaches p < 0.05, and the section says so at length: four of six tasks saturated at 100%, so the effective sample is two tasks, not six.
Pilot 6 model tier × failure shape
Two tasks × three tiers × two arms. On a large mechanical migration both frontier tiers succeeded bare where the cheap model needed the harness, and bare Opus was the cheapest correct result of any configuration tested – $2.43 per success against $3.55 for haiku plus harness – because capability showed up as fewer turns. On the task whose failure mode is omission, nothing substituted for anything: 0 of 18 across every tier and both arms, 17 of them silent, with the full suite green and only the held-out assertions any the wiser.
Pilot 7 epic scale
One feature set decomposed into eight sequentially dependent issues, each delivered by a fresh agent session into the same accumulating repository, so conventions introduced by one issue had to be rediscovered from the code by the next. The agent never sees the other issues, the epic plan, or its own earlier sessions. 18 epics, 144 graded issues, $501.29. The harness fixed the single position where builds break, completely – and changed end-to-end success not at all.
Strict success per position, n=9 per arm. Positions one through six are indistinguishable between arms – the study reports p = 1.000 at each, and publishes no rates for them, so none are drawn here.
i7 cross-cutting migration over existing debt
i8 change established terminal behavior
Issue 7 is where builds break, and the gate runs build as well as test – so it fixed issue 7 outright and touched nothing else. Issue 8 then failed independently at the same rate in both arms, which is enough to fail the whole epic. Six saturated positions, one large gap, and none at the end: a single-task benchmark drawing from those six would conclude the harness does nothing, and one drawing from issue 7 would conclude it is transformative. Both would be describing the same system.
Epic success was 4/9 harnessed against 4/9 bare, identical at every tier, while issue 7 alone went 4/9 to 9/9. Issue 8 then failed independently at the same rate in both arms, which is sufficient to fail the whole epic. Eliminating your most visible failure mode buys nothing a user experiences while a second one remains untouched: a harness evaluated on the failure it was designed to catch will look excellent and change nothing.
Frontier tiers went 4/6 against haiku’s 0/6: a perfect gate cannot make a model capable. And bare Opus was the cheapest, fastest and most reliable configuration tested – $26.37 per epic, 53 minutes, $39.56 per success. That contradicts this study’s own earlier advice: the cheap-model-plus-scaffolding stack recommended by findings 9–10 delivers zero clean epics at any price at this scale. The horizon jump does not merely amplify the harness’s value; it changes which axis matters.
A new module and a new work-item schema failed 0 times in 15. Removing an established terminal behavior and replacing it with a handoff failed 8 of 14, at every tier and in both arms. It also names a harness failure mode: an adversarial reviewer prompted to refute a diff is structurally biased toward the status quo, which is backwards on a behavior-removal issue.
On the omission task the compiler saw no error, the visible suite was green because the agent updated the tests to match its own incomplete change, the done-gate had nothing to block on, and the adversarial reviewer approved the diffs – including for Sonnet and Opus – because a diff shows what changed, not what should have changed and did not. Every signal available to the agent and its harness is a presence-signal; the failure is an absence.
A benchmark authored by one person has to assume it is wrong somewhere and go looking. This one did, in every pilot that bothered, and the results of that search are published beside the results they qualify.
Before any graded run of pilot 7, a full reference implementation of all eight issues was replayed commit by commit, requiring every held-out test to be red on the state before its issue and green after. It caught a specification bug in the author’s own brief: an under-specified clamp returned a range whose minimum (8000) exceeded its maximum (5000). Unfixed, every cell of the grid would have failed issue 7 for correct work, and it would have been published as a finding about model capability. An author-written benchmark has two independent failure modes – the spec can be wrong, and the grader can over-specify – and a reference implementation is the only instrument that separates them.
The p = 0.029 at issue 7 is the study’s first sub-0.05 value in seven pilots, which is precisely when to distrust oneself. All eight positions were tested, not just the interesting one: Bonferroni for eight tests requires p < 0.00625, and Holm–Bonferroni stops at the first step. It survives neither. It is reported as a direction with an independently observed mechanism – the gate’s blocks fire at i7, the bare failures there are build breaks, and every one of 43 harnessed issues ended with a green build against seven bare breaks – and explicitly not as a result.
Pilot 3 recorded four failures. Two were the instrument’s own bugs, found only by autopsying every failure by hand: a holdout that set environment variables in a hook the agent’s idiomatic module-scope initialization ran before, and a re-score that silently used stale build output. Both corrections are append-only events in the journal beside the original records.
vitest transpiles per file and does not typecheck, so the pre-registered definition of success never included the build: three runs across pilots 5–6 left the build broken and passed anyway, one of them through the done-gate and the reviewer. The definition was not changed mid-grid; the build result is recorded on every run and strict success computed retroactively. Separately, a run that came back at $0.16 and 75 seconds with zero files changed turned out to be expired credentials scored as a model failure. Neither problem is visible in a success column.
A "79K-LOC" description of the second target repository counted generated build artifacts as source; the hand-written figure is 31,419 lines. That matters because an earlier section had attributed a difficulty pattern to codebase scale on a 20K-versus-79K gap that is actually 20K versus 31K, so the attribution was downgraded to a conjecture and the affected sections amended rather than silently edited. A number that never gets re-derived is a number nobody has checked.
A no-files-changed alarm on an issue that spent $0.69 over 62 turns was the harness’s bug, not the model’s: metrics were computed from a diff against HEAD, which reports nothing once the agent has committed its own work. That silently zeroed the published diff artifact, the discipline metrics and the tamper check, and would have handed the adversarial reviewer an empty diff. Everything now diffs against a base commit captured before each issue runs, and the grid was rerun from scratch.