Deterministic Gates Beat Model Tier: Opus vs Sonnet in Code Fixes

The thesis behind this pipeline is that the LLM is a minority of the work. Most of what makes an autonomous code fix correct is not the model being clever. It is the scaffolding: the machine-extracted facts from the registry, the behavioral oracle the change has to turn red then green, the mutation gate, the full-suite run, the two Reviewers. If that holds, then changing the model tier should not move the outcome much. A cheaper model plus the gates should land where an expensive one does.

So I ran the same bug fix through four model configurations, end to end (grounding chat, then ticket generation, then the assembly stage, where the Coder and gates run), and watched which gate caught what.

Across every arm, every shipped fix was correct on every real input. Correctness never moved. What moved was which gate had to catch which model’s own failure, and watching that happen is more convincing than asserting the gates would catch it.

Read this as a dated, directional snapshot, not a study. One bug, one target, one plan type, on one day’s version of a pipeline that changes weekly. The numbers go stale the moment the pipeline moves again. The durable assets are the harness, which re-runs in an afternoon, and the findings. There are real confounds, and they come before any numbers.

What was tested: pytest-xdist issue #1335

The bug is pytest-xdist #1335. Under pytest -n2 --dist loadscope, xdist is supposed to keep every test in a module (and class) on one worker so shared fixtures set up once. That breaks for a parametrized test whose parameter value contains ::, a common case being an IPv6 address in an id like test_connect[::1-expected]. Those tests scatter across workers instead of grouping with their siblings.

The cause is in one method, LoadScopeScheduling._split_scope, which derives a test’s scope from its node id by taking everything up to the last :::

def _split_scope(self, nodeid: str) -> str:
    return nodeid.rsplit("::", 1)[0]

A parametrized node id ends with a bracketed parameter id, path/to/test_mod.py::test_name[::1-expected], and when that parameter itself contains ::, the last :: lands inside the brackets instead of at the real module boundary. The correction is a few lines: strip the trailing [...] before splitting on ::. This is exactly the type of small, self-contained fix that lets the model differences show without the bug’s own complexity drowning them out.

The arms

Each arm runs one model on every judgment-bearing LLM stage: the grounding chat, the Brain layer authoring stages, the Coder, the Debugger, and both Reviewers. Trivial classification stages, category labelling and similar, are pinned to the cheap model in every arm, because running a frontier tier on “pick a category” is cost with no signal. Thinking budgets, retries, and Coder flags are held identical across arms. Only the model changes.

• sonnet-4-6, the incumbent. The pipeline’s prompts and thresholds were built against it.
• sonnet-5, a newer generation with a new tokenizer.
• opus-4-8, the expensive tier.
• mixed: sonnet-4-6 everywhere except opus on the three places sonnet has historically wobbled: both Reviewers, and the Brain layer authoring that plans the fix.
• fable-5, the newest frontier tier, held this round. The confounds section explains why.

The cross-arm judges are the model-free gates: the behavioral oracle, confirm-RED, the mutation gate, and the full pytest suite. Those are identical across arms. The Reviewers’ APPROVE verdicts are data about a tier, not the scoring mechanism, so varying them does not turn the comparison into judge-shopping. Correctness is decided by the gates that have no model in them.

The confounds, before the numbers

Home-field advantage is large. Every prompt and threshold in the pipeline was tuned against sonnet-4-6. A different model running those prompts is on someone else’s field. If a frontier model underperforms, prompt-fit is a live alternative explanation, not a tier ranking. One of the findings below is the incumbent’s own prompts failing to steer a model to a valid answer, so this is not a hypothetical.

This run is not a controlled experiment, and it cannot be. A controlled comparison needs a frozen, finished pipeline; this one is mid-development by design. To get three clean completions per sonnet arm for a completion table, I re-rolled past the non-clean runs. That is a cherry-pick. Why the non-clean runs happened matters more than the manufactured 3/3.

N is small: three per sonnet arm, one for mixed. These are existence proofs and crude rates.

Fable is held because fable-5 is the newest generation, which makes it the most home-field-confounded arm of all. If it churned, there would be no way to separate “the frontier model did not help” from “prompts two generations old do not fit it.” A confounded datapoint presented as a clean one is worse than none. An earlier wave on an older pin ran fable to a full completion record; I cite that directionally rather than manufacture a muddy new one.

The results

The result that matters is not in the table. Across every arm and every roll, including the two degrades and the one termination, the pipeline never committed a broken fix. Every fix that shipped derived the correct scope for every node id pytest can actually produce. The tier differences show up in cost, in how much the model churned, and in which gate had to catch the wobble. They do not show up in correctness.

The gates absorbed the model, a different gate each time

This is the thesis, and it held in the most useful way it could: each arm’s model failed differently, and a different piece of scaffolding caught each failure.

opus planned a fragile fix on one run. A pre-flight check validated a plan that cut the node id at the first [:

# opus's plan, validated by the pipeline's plan-check on attempt one
structural = nodeid.split("[", 1)[0]
if "::" in structural:
    return structural.rsplit("::", 1)[0]
return nodeid

That mishandles a [ appearing earlier in the node id than the real parameter, the same class of bug the bare CLI agent’s roll 3 would independently ship later in this post. The check’s own test cases never included that discriminating input, so the plan validated clean.

The behavioral oracle downstream did include it. The Coder shipped the plan as written, and the end-of-ticket test came back 14 passed, 4 failed, every failure a node id with a stray [ before the real parameter. The Debugger fired next and was handed a broken manifest: an 11 passed test-output block pasted against a constraint recording 14 failed. It refused to guess at a fix from a contradiction it couldn’t resolve, and asked for the real failing output instead. The manifest was corrected, the algorithm re-authored, and the Coder converged to the bracket-depth-aware form shown in the robustness section below. Final oracle: 18 passed, 0 failed. Mutation gate: 6 killed, 0 survived. Full suite: 245 passed. This is the run behind the $3.08 outlier already noted in the results table above; the stage-by-stage breakdown in the cost appendix below shows the repair machinery, not the original mistake, is where that money went.

sonnet-5 shipped a fix whose committed tests did not kill one derived mutant. The mutation gate searched a generated set of discriminating inputs, found one that separates the mutant from the committed code, pinned it as a new test case, and re-ran green. No LLM was involved in that repair.

sonnet-4-6 could not author a valid behavioral oracle on two rolls. Once its behavioral scenarios collected zero tests when rendered; once the plan it authored would not converge. Both times the pipeline refused the weak oracle and fell back, loudly, to the valid unit-level oracle, flagging the run as degraded: no wrong-green, no broken code committed. A weaker authoring pass produced a caught result, not a worse one.

mixed passed cleanly, and its one surviving mutant was auto-repaired by the same machine search as sonnet-5’s.

That is what “the model is a minority of the work” looks like when you watch it happen rather than assert it from a cost curve.

The intelligence that made these fixes correct was in the gate coverage, not the tier.

The control arm: the same bug, no pipeline at all

Every arm above varies the model inside the pipeline. None of them answers the question a reader should be asking after the confounds above: what happens if you skip the scaffolding entirely and hand the ticket to a competent agent with a shell?

So I ran that. The same ticket, byte-identical apart from one added line asking for tests, given to a bare CLI agent on the strongest model available. No registry, no manifest, no gates. It reads the repo, greps, edits, runs pytest, and decides for itself when it is done. Five rolls, no re-rolls, every roll reported.

Four clean rolls, one that wasn’t

Four of the five produced a fix I would merge, found the project’s changelog-fragment convention and pytester idiom unprompted, and appended to the existing test files rather than clobbering them. The pipeline’s four arms ranged $0.74 to $1.70; the nearest apples-to-apples point, opus at $1.54 to $1.70, since both ran the strongest model available, still costs more than every bare-agent roll.

Roll 3 shipped the identical failure mode as opus’s fragile plan earlier in this post. Put a bracket in a directory name, legal if uncommon, and three different modules, one of them class-based, merge into a single work unit, the ticket’s explicit do-not-regress constraint, broken:

# roll 3's fix, on a path containing a bracket
"p[ath]/test_mod.py::test_x[::1]"          # scope -> "p"
"p[ath]/test_mod.py::TestC::test_x[::1]"   # scope -> "p"
"p[ath]/other_mod.py::test_y[::1]"         # scope -> "p"

Nothing in that run saw it. The full suite passed, 229 tests green. It wrote a changelog entry. Every check the agent ran on itself came back clean, and the fix was wrong. The agent has no red-before-green gate of its own, so I checked by hand afterward: reverted to the pre-fix code and reran its tests. Three failed, so they weren’t vacuous, they genuinely pinned behaviour. That kills a comfortable assumption: that a test failing before the fix and passing after is a sufficient bar. It clears that bar and is still wrong, because what it lacks is a case with a bracket in the path, and no amount of red-before-green tells you which cases are missing.

The pipeline caught and repaired the opus version of this exact bug for $3.08. Nothing caught this one.

Whether the $2.47 difference between a fix that might be silently wrong and one guaranteed caught is worth paying is the reader’s call, not mine.

What would have caught it

The pipeline’s answer is not a better test-writing prompt. It constructs node ids mechanically that probe every place the [ delimiter could legally recur, and requires the proposed rule to agree with the pre-fix code everywhere the ticket did not license a change. Roll 3’s shipped implementation diverges on two of fourteen such inputs; the correct version diverges on none. Seven recorded runs on this ticket authored the same first-bracket over-reach across three model tiers, and the check caught every one:

05:29:07  divergence check on '_split_scope': the proposed rule changes
          behavior on an input outside the ticket's scope:
          'p[ath/to/test_mod.py::TestC::test_name'
05:29:42  passed after one re-author pass

Thirty-five seconds, before a single line of implementation was written. Six of the seven were repaired automatically; the seventh degraded loudly to a weaker oracle instead, the failure documented earlier in this post. None of the seven shipped a broken fix. The counterexample differs every run, nobody wrote those cases down; they fall out of the construction, which is the only reason this generalises beyond the one bug I happened to be looking at.

The bare agent was cheaper and faster, and right four times out of five. The fifth time it was wrong in a way that nothing it could run on itself would reveal.

It is an existence proof, not a rate: five rolls cannot tell you how often this happens. The pipeline’s evidence is gen-stage, the check rejected a bad plan before code existed, not that it caught bad code in review.

What this run says about the thesis

Directionally it holds. Across four model configurations on this bug, correctness did not move: every committed fix was right on every real input, and the gates decided that, not the tier. The tier bought smoothness, not correctness; stronger models churned and degraded less, cheaper models leaned on the gates more, and they all landed in the same place because the gates made them. Targeted frontier placement, opus on the one stage that could not converge, beat a bigger model everywhere on both reliability and cost. The mechanics behind that claim, what things cost, what “robust enough” means, and where the pipeline’s own gates still need work, are below.

None of that is a definitive ranking, and it is not meant to be. It is one dated data point for the claim that the model is a minority of the work, with the added benefit that you can see which minority, and which gate covered the rest.

Appendix: cost

Three things this pipeline is not

Three concessions before the cost breakdown below.

It is not price-optimised, and the biggest line is not the model. Splitting the full chain’s opus spend by billing category: 63% of it is prompt-cache writes. The chain makes several separate stage invocations, each starting from cold context and paying the cache-write premium; cache writes outweigh cache reads roughly five to one, and output tokens, the actual authored work, are only 37% of the total, none of it tuned.

You pay that premium once per stage, and it is the price of the isolation. The property that keeps each stage seeing only what it needs, with no context bleed between them, is the same property that stops any stage reusing another’s cache. A single long-running agent on the same ticket inverts the ratio completely: cache reads outweigh cache writes roughly thirty to one, at a tenth of the input rate. Its cache reads are 42% of its spend where the pipeline’s are 1%. Stage isolation is a real architectural cost, measured here against a single-session control. Genuine extra work is the smaller half of the gap. The pipeline generates 2.3x the output tokens, not 2.3x the dollars.

The pipeline is over-engineered for a repo this size. pytest-xdist’s source and tests together are roughly 74,000 tokens. A current frontier model’s context window is a million. The registry, the surgical reads, the manifest that tells the Coder exactly which file and symbol to open, all of that exists because a real codebase does not fit in a context window. This one fits thirteen times over. On this fixture that machinery is solving a problem the fixture does not have, and any honest reading of the numbers below has to start there.

Where the money goes: the Brain layer’s share, when assembly goes smoothly

Splitting the full-chain billed cost by stage group makes a pattern fall out. Chat and ticket generation are the Brain layer; the assembly stage is the Coder and the gates. These are actual per-call billed dollars, cache-discounted.

When assembly goes smoothly, the Brain layer dominates: authoring costs 1.1 to 2.2 times the assembly across every clean run. Within the Brain layer, the money is in authoring the behavioral oracle and the ticket narrative; formalizing the fix plan is cheap by comparison.

When assembly goes rough, its cost balloons and flips the ratio. The opus outlier at $3.08 is the run whose fragile first attempt triggered a Debugger call plus a repair pass, two Coder calls, a Debugger, and a re-authored spec. Its assembly cost hit $1.91, three times a clean run, dropping the ratio to 0.5 : 1. Assembly is the cheap part only when the code is right the first time. When it is not, the repair machinery is where the dollars land. For anyone costing an autonomous pipeline against engineer-hours, the takeaway is that the variance lives in the recovery path, not the happy path.

On a right-first-time fix, the single most expensive stage in assembly is usually the Test Reviewer, not the Coder: on one opus run, Test Reviewer at $0.33 against Coder at $0.18. Verifying the code costs more than writing it. For a pipeline whose whole claim is that correctness comes from checking rather than generation, that is the cost profile you would expect.

The smart money is targeted placement, not a bigger model everywhere

The mixed arm is the most useful result of the run. It is sonnet-4-6 everywhere except opus on both Reviewers and the Brain layer authoring that plans the fix, the three places the cheap model wobbles.

On its roll it completed clean where sonnet-4-6 degraded on two of five. Opus authored a valid plan on the first pass, so the behavioral oracle validated and the run never fell back. The mixed ticket-generation cost $0.40, the cheapest of any arm, cheaper than all-sonnet-4-6 at $0.54 to $0.57. sonnet-4-6 needed two re-author cycles per generation to converge; opus got it right the first time and skipped the churn. Spending on the one stage that could not converge made generation both more reliable and cheaper.

Full-chain, mixed came in at $1.05, opus-grade generation reliability at a third less than full opus. That is n=1 and directional. It is also the cleanest answer to “where does spending on the frontier tier actually pay”: not everywhere, only where the cheap model cannot converge.

Appendix: robustness

Minimal-correct beats maximal-robust, and the pipeline knows it

I built a 16-case set to grade the committed fixes for robustness. Most runs scored 15 of 16. The one input they missed was m.py::TestC::test_x[a::b[c]], a parameter value containing both a :: and a nested bracket.

My first read was to call those fixes fragile. That read is wrong. m.py::TestC::test_x[a::b[c]] is not a real pytest node id; pytest does not produce parameter ids with that structure. The 15-of-16 fixes used a straightforward right-side search for the last [. They handle every input that can occur, including the one a maintainer might reasonably worry about, a [ inside a file path. The 16-of-16 fixes used a bracket-depth counter, extra machinery for a case that will not happen.

# what most Coders shipped: correct on every real node id
if nodeid.endswith("]"):
    nodeid = nodeid[: nodeid.rfind("[")]
return nodeid.rsplit("::", 1)[0]

# what a few shipped: a bracket-depth counter to match nested brackets,
# correct too, but machinery for a parameter id pytest cannot produce
if nodeid.endswith("]"):
    depth = 0
    for i in range(len(nodeid) - 1, -1, -1):
        if nodeid[i] == "]":
            depth += 1
        elif nodeid[i] == "[":
            depth -= 1
            if depth == 0:
                nodeid = nodeid[:i]
                break
return nodeid.rsplit("::", 1)[0]

The first version is not less robust than the second. It is leaner, and for this problem leaner is correct. “Be more robust” has no floor: there is always another pathological string, so you would play whack-a-mole forever and grow the codebase against inputs that cannot occur.

The pipeline’s bias runs the other way. Coders repeatedly simplified the robust reference down to the minimal correct form, and nothing was wrong with that. The one genuinely fragile version, the one that failed a plausible input, is the one the gates caught. The line is drawn in the right place: reject wrong-on-real-inputs, do not mandate robustness against the impossible.

The Brain layer produces a reference version of the fix, but that reference exists to check the plan against the pre-fix behavior, not as a script the Coder must transcribe. The Brain layer decides what to build and authors the oracle that pins it; the Coder decides how to write it. A fix is only as robust as the oracle makes it. If you want a behavior guaranteed, you put a case for it in the oracle, rather than trusting the builder to remember it. This is the Data Path Principle (DPP) applied to craft: a decision the Brain layer wants enforced becomes a machine-checkable case, never an instruction the builder is trusted to follow.

Robustness lives in the oracle, not in a mandate on the builder.

Appendix: pipeline gaps

Where the pipeline itself was wrong

The gates mostly did their job. One did not, in an instructive way.

sonnet-5’s one termination, the non-clean roll behind its 3-clean-of-4 completion rate above, filed as Gap #724 in the architecture-gaps document, was not a model failure. The Coder shipped correct code, 16 of 16 on the robustness set, code that would have passed everything. It included a redundant guard, an endswith("]") check that a default already covered. The mutation gate flipped that guard’s string literal to empty and produced a mutant that behaves identically to the real code on every realistic input, an equivalent mutant. Nothing could kill it, the machine search correctly found no killer, and the gate then failed the run instead of recognising the mutant as unkillable.

This is the classic equivalent-mutant false positive from mutation testing, and here it runs inverted against code quality: the more defensive the model’s code, the more redundant guards it carries, the more equivalent mutants it generates, the more the gate false-fails correct code. It is loud and pre-commit, never a false GREEN, so it fails safe. It is still a gate-calibration bug, and it is filed as one.

Keep this apart from the catches in the gates section above. Those were the gates stopping real weakness. This was a gate erring on correct work. Both belong in the post; they are not the same category, and collapsing them would flatter the pipeline.

The benchmark as a fuzzer for the pipeline’s own contracts

Every model swap this run surfaced latent pipeline defects that months of same-model runs never did. Five are now filed in the architecture-gaps document: a machine-pinned test that does not match its file’s construction idiom; a Debugger handed the wrong test output in its brief; a pre-flight check that green-lights a fragile plan; the equivalent-mutant false-fail above; and a re-author cycle that feeds a model the same feedback and gets the same wrong answer back twice. None shipped a bad fix. Every one was refused loudly or caught downstream. They are only visible when you run a distribution the pipeline was not tuned against. Swapping the model is, among other things, a fuzzer for every implicit contract between stages.

The last of those is also the sharpest evidence for the home-field caveat. It was the incumbent model, sonnet-4-6, failing to converge under the pipeline’s own prompts, tuned to its own model, unable to steer it off a repeated mistake. Prompt-fit to a model family buys a great deal. It does not buy guaranteed convergence.

This run targeted pytest-xdist, a public open-source project, for a controlled model-tier benchmark rather than the pipeline’s usual ~100k line TypeScript monorepo. Every fix described here is a committed diff awaiting PR review, not a merged change. Pipeline pin a27a209, run 2026-07-24. One bug, one target, one plan type. Not a controlled study. Still R&D.