Opus vs Sonnet: Deterministic Gates Beat Model Tier in Code Fixes

The thesis behind this pipeline is that the LLM is a minority of the work. Most of what makes an autonomous code fix correct is not the model being clever. It is the scaffolding: the machine-extracted facts from the registry, the behavioral oracle the change has to turn red then green, the mutation gate, the full-suite run, the two Reviewers. If that holds, then changing the model tier should not move the outcome much. A cheaper model plus the gates should land where an expensive one does.

So I ran the same bug fix through four model configurations, end to end (grounding chat, then ticket generation, then the assembly stage, where the Coder and gates run), and watched which gate caught what.

Read this as a dated, directional snapshot, not a study. One bug, one target, one plan type, on one day’s version of a pipeline that changes weekly. The numbers go stale the moment the pipeline moves again. The durable assets are the harness, which re-runs in an afternoon, and the findings. There are real confounds, and they come before any numbers.

What was tested: pytest-xdist issue #1335

The bug is pytest-xdist #1335. Under pytest -n2 --dist loadscope, xdist is supposed to keep every test in a module (and class) on one worker so shared fixtures set up once. That breaks for a parametrized test whose parameter value contains ::, a common case being an IPv6 address in an id like test_connect[::1-expected]. Those tests scatter across workers instead of grouping with their siblings.

The cause is in one method, LoadScopeScheduling._split_scope, which derives a test’s scope from its node id by taking everything up to the last :::

def _split_scope(self, nodeid: str) -> str:
    return nodeid.rsplit("::", 1)[0]

A parametrized node id ends with a bracketed parameter id, path/to/test_mod.py::test_name[::1-expected], and when that parameter itself contains ::, the last :: lands inside the brackets instead of at the real module boundary. The correction is a few lines: strip the trailing [...] before splitting on ::. This is exactly the type of small, self-contained fix that lets the model differences show without the bug’s own complexity drowning them out.

The arms

Each arm runs one model on every judgment-bearing LLM stage: the grounding chat, the Brain layer authoring stages, the Coder, the Debugger, and both Reviewers. Trivial classification stages, category labelling and similar, are pinned to the cheap model in every arm, because running a frontier tier on “pick a category” is cost with no signal. Thinking budgets, retries, and Coder flags are held identical across arms. Only the model changes.

• sonnet-4-6, the incumbent. The pipeline’s prompts and thresholds were built against it.
• sonnet-5, a newer generation with a new tokenizer.
• opus-4-8, the expensive tier.
• mixed: sonnet-4-6 everywhere except opus on the three places sonnet has historically wobbled: both Reviewers, and the Brain layer authoring that plans the fix.
• fable-5, the newest frontier tier, held this round. The confounds section explains why.

The cross-arm judges are the model-free gates: the behavioral oracle, confirm-RED, the mutation gate, and the full pytest suite. Those are identical across arms. The Reviewers’ APPROVE verdicts are data about a tier, not the scoring mechanism, so varying them does not turn the comparison into judge-shopping. Correctness is decided by the gates that have no model in them.

The confounds, before the numbers

Home-field advantage is large. Every prompt and threshold in the pipeline was tuned against sonnet-4-6. A different model running those prompts is on someone else’s field. If a frontier model underperforms, prompt-fit is a live alternative explanation, not a tier ranking. One of the findings below is the incumbent’s own prompts failing to steer a model to a valid answer, so this is not a hypothetical.

This run is not a controlled experiment, and it cannot be. A controlled comparison needs a frozen, finished pipeline; this one is mid-development by design. To get three clean completions per sonnet arm for a completion table, I re-rolled past the non-clean runs. That is a cherry-pick. Why the non-clean runs happened matters more than the manufactured 3/3.

N is small: three per sonnet arm, one for mixed. These are existence proofs and crude rates.

Fable is held because fable-5 is the newest generation, which makes it the most home-field-confounded arm of all. If it churned, there would be no way to separate “the frontier model did not help” from “prompts two generations old do not fit it.” A confounded datapoint presented as a clean one is worse than none. An earlier wave on an older pin ran fable to a full completion record; I cite that directionally rather than manufacture a muddy new one.

Three things this pipeline is not

Before any cost number appears, three concessions. They are measurements, not modesty, and the last section of this post only means something if you have read them.

It is not price-optimised, and the biggest line is not the model. Splitting the full chain’s opus spend by billing category: 63% of it is prompt-cache writes. The chain makes ten separate stage invocations, each starting from cold context and paying the cache-write premium; cache writes outweigh cache reads roughly five to one, and output tokens, the actual authored work, are only 37% of the total, none of it tuned.

You pay that premium once per stage, and it is the price of the isolation. The property that keeps each stage seeing only what it needs, with no context bleed between them, is the same property that stops any stage reusing another’s cache. A single long-running agent on the same ticket inverts the ratio completely: cache reads outweigh cache writes roughly thirty to one, at a tenth of the input rate. Its cache reads are 42% of its spend where the pipeline’s are 1%. Stage isolation is a real architectural cost, and I had not priced it until I measured against a single-session control. Genuine extra work is the smaller half of the gap. The pipeline generates 2.3x the output tokens, not 2.3x the dollars.

The pipeline is over-engineered for a repo this size. pytest-xdist’s source and tests together are roughly 74,000 tokens. A current frontier model’s context window is a million. The registry, the surgical reads, the manifest that tells the Coder exactly which file and symbol to open, all of that exists because a real codebase does not fit in a context window. This one fits thirteen times over. On this fixture that machinery is solving a problem the fixture does not have, and any honest reading of the numbers below has to start there.

The results

The result that matters is not in the table. Across every arm and every roll, including the two degrades and the one termination, the pipeline never committed a broken fix. Every fix that shipped derived the correct scope for every node id pytest can actually produce. The tier differences show up in cost, in how much the model churned, and in which gate had to catch the wobble. They do not show up in correctness.

The gates absorbed the model, a different gate each time

This is the thesis, and it held in the most useful way it could: each arm’s model failed differently, and a different piece of scaffolding caught each failure.

opus planned a fragile fix on one run. It stripped from the first [ in the node id, which mishandles a [ that appears in the file path itself. The pipeline’s own check on that plan passed it, because the inputs it checked against did not include that discriminating case. The behavioral oracle downstream did include it: the end-of-ticket test came back 14 passed, 4 failed, the Debugger fired, a repair pass ran, and the Coder converged to the correct form, catching and repairing the fragile plan.

sonnet-5 shipped a fix whose committed tests did not kill one derived mutant. The mutation gate searched a generated set of discriminating inputs, found one that separates the mutant from the committed code, pinned it as a new test case, and re-ran green. No LLM was involved in that repair.

sonnet-4-6 could not author a valid behavioral oracle on two rolls. Once its behavioral scenarios collected zero tests when rendered; once the plan it authored would not converge. Both times the pipeline refused the weak oracle and fell back, loudly, to the valid unit-level oracle, flagging the run as degraded: no wrong-green, no broken code committed. A weaker authoring pass produced a caught result, not a worse one.

mixed passed cleanly, and its one surviving mutant was auto-repaired by the same machine search as sonnet-5’s.

That is what “the model is a minority of the work” looks like when you watch it happen rather than assert it from a cost curve.

The intelligence that made these fixes correct was in the gate coverage, not the tier.

Minimal-correct beats maximal-robust, and the pipeline knows it

I built a 16-case set to grade the committed fixes for robustness. Most runs scored 15 of 16. The one input they missed was m.py::TestC::test_x[a::b[c]], a parameter value containing both a :: and a nested bracket.

My first read was to call those fixes fragile. That read is wrong. m.py::TestC::test_x[a::b[c]] is not a real pytest node id; pytest does not produce parameter ids with that structure. The 15-of-16 fixes used a straightforward right-side search for the last [. They handle every input that can occur, including the one a maintainer might reasonably worry about, a [ inside a file path. The 16-of-16 fixes used a bracket-depth counter, extra machinery for a case that will not happen.

# what most Coders shipped: correct on every real node id
if nodeid.endswith("]"):
    nodeid = nodeid[: nodeid.rfind("[")]
return nodeid.rsplit("::", 1)[0]

# what a few shipped: a bracket-depth counter to match nested brackets,
# correct too, but machinery for a parameter id pytest cannot produce
if nodeid.endswith("]"):
    depth = 0
    for i in range(len(nodeid) - 1, -1, -1):
        if nodeid[i] == "]":
            depth += 1
        elif nodeid[i] == "[":
            depth -= 1
            if depth == 0:
                nodeid = nodeid[:i]
                break
return nodeid.rsplit("::", 1)[0]

The first version is not less robust than the second. It is leaner, and for this problem leaner is correct.

Which is what you do not want from a model writing your code. “Be more robust” has no floor: there is always another pathological string, so you would play whack-a-mole forever and grow the codebase against inputs that cannot occur. The pipeline’s bias runs the other way. Coders repeatedly simplified the robust reference down to the minimal correct form, and nothing was wrong with that. The one genuinely fragile version, the one that failed a plausible input, is the one the gates caught. The line is drawn in the right place: reject wrong-on-real-inputs, do not mandate robustness against the impossible.

There is an architecture point under this. The Brain layer produces a reference version of the fix, but that reference exists to check the plan against the pre-fix behavior, not as a script the Coder must transcribe. The Brain layer decides what to build and authors the oracle that pins it; the Coder decides how to write it. A fix is only as robust as the oracle makes it. If you want a behavior guaranteed, you put a case for it in the oracle. You do not force the Coder to copy the reference, and you do not add oracle cases for inputs that cannot happen. This is the Data Path Principle (DPP) applied to craft: a decision the Brain layer wants enforced becomes a machine-checkable case, never an instruction the builder is trusted to follow.

Robustness lives in the oracle, not in a mandate on the builder.

Where the money goes: the Brain layer’s share, when assembly goes smoothly

Splitting the full-chain billed cost by stage group makes a pattern fall out. Chat and ticket generation are the Brain layer; the assembly stage is the Coder and the gates. These are actual per-call billed dollars, cache-discounted.

When assembly goes smoothly, the Brain layer dominates: authoring costs 1.1 to 2.2 times the assembly across every clean run. Within the Brain layer, the money is in authoring the behavioral oracle and the ticket narrative; formalizing the fix plan is cheap by comparison.

When assembly goes rough, its cost balloons and flips the ratio. The opus outlier at $3.08 is the run whose fragile first attempt triggered a Debugger call plus a repair pass, two Coder calls, a Debugger, and a re-authored spec. Its assembly cost hit $1.91, three times a clean run, dropping the ratio to 0.5 : 1. Assembly is the cheap part only when the code is right the first time. When it is not, the repair machinery is where the dollars land. For anyone costing an autonomous pipeline against engineer-hours, the takeaway is that the variance lives in the recovery path, not the happy path.

On a right-first-time fix, the single most expensive stage in assembly is usually the Test Reviewer, not the Coder: on one opus run, Test Reviewer at $0.33 against Coder at $0.18. Verifying the code costs more than writing it. For a pipeline whose whole claim is that correctness comes from checking rather than generation, that is the cost profile you would expect.

The smart money is targeted placement, not a bigger model everywhere

The mixed arm is the most useful result of the run. It is sonnet-4-6 everywhere except opus on both Reviewers and the Brain layer authoring that plans the fix, the three places the cheap model wobbles.

On its roll it completed clean where sonnet-4-6 degraded on two of five. Opus authored a valid plan on the first pass, so the behavioral oracle validated and the run never fell back. The mixed ticket-generation cost $0.40, the cheapest of any arm, cheaper than all-sonnet-4-6 at $0.54 to $0.57. sonnet-4-6 needed two re-author cycles per generation to converge; opus got it right the first time and skipped the churn. Spending on the one stage that could not converge made generation both more reliable and cheaper.

Full-chain, mixed came in at $1.05, opus-grade generation reliability at a third less than full opus. That is n=1 and directional. It is also the cleanest answer to “where does spending on the frontier tier actually pay”: not everywhere, only where the cheap model cannot converge.

Where the pipeline itself was wrong

The gates mostly did their job. One did not, in an instructive way.

sonnet-5’s single non-completion, filed as Gap #724 in the architecture-gaps document, was not a model failure. The Coder shipped correct code, 16 of 16 on the robustness set, code that would have passed everything. It included a redundant guard, an endswith("]") check that a default already covered. The mutation gate flipped that guard’s string literal to empty and produced a mutant that behaves identically to the real code on every realistic input, an equivalent mutant. Nothing could kill it, the machine search correctly found no killer, and the gate then failed the run instead of recognising the mutant as unkillable.

This is the classic equivalent-mutant false positive from mutation testing, and here it runs inverted against code quality: the more defensive the model’s code, the more redundant guards it carries, the more equivalent mutants it generates, the more the gate false-fails correct code. It is loud and pre-commit, never a false GREEN, so it fails safe. It is still a gate-calibration bug, and it is filed as one.

Keep this apart from the catches above. Those were the gates stopping real weakness. This was a gate erring on correct work. Both belong in the post; they are not the same category, and collapsing them would flatter the pipeline.

The benchmark as a fuzzer for the pipeline’s own contracts

Every model swap this run surfaced latent pipeline defects that months of same-model runs never did. Five are now filed in the architecture-gaps document: a machine-pinned test that does not match its file’s construction idiom; a Debugger handed the wrong test output in its brief; a pre-flight check that green-lights a fragile plan; the equivalent-mutant false-fail above; and a re-author cycle that feeds a model the same feedback and gets the same wrong answer back twice. None shipped a bad fix. Every one was refused loudly or caught downstream. They are only visible when you run a distribution the pipeline was not tuned against. Swapping the model is, among other things, a fuzzer for every implicit contract between stages.

The last of those is also the sharpest evidence for the home-field caveat. It was the incumbent model, sonnet-4-6, failing to converge under the pipeline’s own prompts, tuned to its own model, unable to steer it off a repeated mistake. Prompt-fit to a model family buys a great deal. It does not buy guaranteed convergence.

The control arm: the same bug, no pipeline at all

Every arm above varies the model inside the pipeline. None of them answers the question a reader should be asking after the three concessions: what happens if you skip the scaffolding entirely and hand the ticket to a competent agent with a shell?

So I ran that. The same ticket, byte-identical apart from one added line asking for tests, given to a bare CLI agent on the strongest model available. No registry, no manifest, no gates. It reads the repo, greps, edits, runs pytest, and decides for itself when it is done. Five rolls, no re-rolls, every roll reported.

Four of the five produced a fix I would merge. They were fast and they cost between $0.58 and $1.11, against $1.54 for the full pipeline chain. Every one of them found the project’s changelog-fragment convention unprompted, used the surrounding suite’s pytester idiom, and appended to the existing test files rather than clobbering them. On a fixture this size, with a ticket this well-specified, the bare agent is good.

Roll 3 stripped the parametrization by searching for the first [ in the node id. On any path without a bracket that is indistinguishable from the correct fix. Put a bracket in a directory name, legal, if uncommon, and the scope collapses below module level:

# roll 3's fix, on a path containing a bracket
"p[ath]/test_mod.py::test_x[::1]"          # scope -> "p"
"p[ath]/test_mod.py::TestC::test_x[::1]"   # scope -> "p"
"p[ath]/other_mod.py::test_y[::1]"         # scope -> "p"

Three different modules, one of them class-based, merged into a single work unit. That is the ticket’s explicit do-not-regress constraint, broken, and it inverts the reported bug: instead of scattering a module’s tests across workers it merges unrelated ones together, and the fifth roll is the reason the pipeline exists.

Nothing in that run saw it. The full suite passed, 229 tests green. It wrote tests, and the tests were real: three of them fail against the pre-fix code, so they genuinely pin behaviour rather than passing vacuously. It wrote a changelog entry. Every check the agent performed on itself came back clean, and the fix was wrong.

That kills a comfortable assumption, including one I held: that “did the agent write a test that fails before the fix” is a sufficient bar. Roll 3 clears that bar. What roll 3 does not have is a case in its test set with a bracket in the path, and no amount of red-before-green tells you which cases are missing. Sufficiency is about coverage of the discriminating input, and an agent grading its own work cannot know what it did not think of.

The pipeline’s answer to this is not a better test-writing prompt. It generates the discriminating inputs mechanically: it constructs node ids that probe every place the [ delimiter could legally recur, and requires that the proposed rule agree with the pre-fix code everywhere the ticket did not license a change. Run roll 3’s shipped implementation through that generator and it diverges on two of fourteen inputs. Run the correct version and it diverges on none.

That is not hypothetical. Seven recorded runs on this ticket authored the same first-bracket over-reach, across three different model tiers, and the check caught every one:

05:29:07  divergence check on '_split_scope': the proposed rule changes
          behavior on an input outside the ticket's scope:
          'p[ath/to/test_mod.py::TestC::test_name'
05:29:42  passed after one re-author pass

Thirty-five seconds, before a single line of implementation was written. Six of the seven were repaired automatically that way. The seventh could not be steered off its answer and refused to ship it, degrading loudly to a weaker oracle instead, the failure documented earlier in this post, and none of the seven shipped a broken fix.

The counterexample is different in every run: p[ath/to/..., p[ath/test_mod.py::test_connect, once even a bracket spliced into the :: separator itself. Nobody wrote those cases down. They fall out of the construction, which is the only reason this generalises beyond the one bug I happened to be looking at.

The bare agent was cheaper and faster, and right four times out of five. The fifth time it was wrong in a way that nothing it could run on itself would reveal.

It is an existence proof, not a rate: five rolls cannot tell you how often a frontier agent ships a silent defect, and I make no claim that it is one in five. The pipeline evidence is gen-stage: the check rejected a bad plan before code existed, rather than catching bad code in review. And the concessions above still stand: this chain costs roughly twice a single session, most of that premium is cache writes rather than thinking, and on a repo that fits comfortably in a context window most of the machinery is answering a question the fixture never asks.

None of which changes the one thing I care about. On this ticket, the unscaffolded agent shipped a broken commit and could not have known. The scaffolded one has never let that class through.

What this run says about the thesis

Directionally it holds. Across four model configurations on this bug, correctness did not move: every committed fix was right on every real input, and the gates decided that, not the tier. The tier bought smoothness, not correctness; stronger models churned and degraded less, cheaper models leaned on the gates more, and they all landed in the same place because the gates made them. The expensive work is the authoring and the checking, and it is cheap only when the code is right the first time. Targeted frontier placement, opus on the one stage that could not converge, beat a bigger model everywhere on both reliability and cost.

None of that is a definitive ranking, and it is not meant to be. It is one dated data point for the claim that the model is a minority of the work, with the added benefit that you can see which minority, and which gate covered the rest.

This run targeted pytest-xdist, a public open-source project, for a controlled model-tier benchmark rather than the pipeline’s usual ~100k line TypeScript monorepo. Every fix described here is a committed diff awaiting PR review, not a merged change. Pipeline pin a27a209, run 2026-07-24. One bug, one target, one plan type. Not a controlled study. Still R&D.