Before July, every LLM call in this system had only ever been served by one model. Twenty-four consecutive generations of a plan artifact came back green, the unit suite was green, and the pipeline had committed correct fixes against a real open-source repository. Then I swapped in a different model from the same provider, not a jump across vendors, and the first run hit a defect in my code before any quality comparison happened at all.
That was not bad luck: repetition with a single model explores depth, not breadth. A model’s structural authoring habits are close to deterministic at fixed prompts and settings, so twenty-four more runs on the incumbent would never have sampled the convention that broke things. Swapping the model samples a different distribution across every boundary at once, which makes it a fuzzer for the code that consumes model output, an idea the model-tier benchmark post touched on in passing; this post is the full record.
None of this is specific to code generation or multi-stage systems. If you parse a model’s JSON, dispatch on a field it fills in, or template its text into a downstream call, the correctness of that code depends on a distribution you have only ever sampled from one model.
What a model swap actually tests
Wherever model output meets your code there is a contract: each stage promises the next a particular shape of output, and downstream code trusts that promise without checking it. Most of these contracts are unvalidated and invisible by construction, because every run on the incumbent model is simultaneously the test of the harness and the calibration of what valid output looks like. Nobody writes the assumption down, because the only distribution anyone has ever observed sits comfortably inside it.
A model swap is the cheapest way I have found to sample outside that distribution.
It costs one config field and exercises every handoff at once, including ones you would never think to write a property test for.
The better output is the one that breaks you
The first two findings had the same anatomy: the newer model produced a more precise, more correct output than the incumbent, and that precision is what broke the pipeline.
During planning, the pipeline asked for a call to the edited symbol with a crafted input. The incumbent always took the surface reading, one bare argument; the new model modelled the full call signature and added an explicit empty receiver slot ahead of it. Only the incumbent’s form had ever reached the harness, so the extra argument forced a wrong guess about where the real input sat, and three rounds burned defending a rule that had been correct since round zero.
A third model hard-blocked the grounding chat on its very first call: it was the only one of the three that identified the correct source file from the issue prose and volunteered that detail where the others habitually left it blank. The chat checked the detail against what it already knew, found no match, and abandoned the correctly resolved target instead of narrowing it.
Both are a Data Path Principle violation, not a parsing bug: letting free text veto machine-extracted resolution inverts the trust hierarchy, an inversion that had sat in the code for roughly a hundred sessions because no incumbent model ever had anything to volunteer there. In both cases, the stronger output broke the pipeline because the pipeline had quietly encoded one model’s laziness as a precondition.
Four disjoint sets of defects
The pattern repeated at every scale. Across two full benchmark waves, model swaps produced more than twenty filed rows in the architecture-gaps document. Two independent swaps each found four defects, none reachable from the other’s findings, ranging from a generated test guard that collected zero tests to a mutation gate that held both a surviving mutant and the fix for it without the search to connect them. A third model found the Coder being offered an operation whose driver only exists for TypeScript, because the menu offered to a builder was hardcoded rather than derived from actual capability. A fourth swap caught a benchmark design claim that had never been true outside the incumbent arms: three stages assumed pinned to a cheap model were silently running at full price, because a chat session resolved its model config once for every sub-call it made.
The diagram has the full set. What matters is that it kept happening: any invariant your system asserts about a code path only one input distribution has ever traversed is unverified, whether or not it happens to be true.
Fixtures encode the distribution you’ve already seen
The same blind spot shows up again and again, and it is the most transferable lesson here: fixtures written to match one model’s output only ever prove that model works, not the code. The sharpest example was a fix for a model-swap bug itself. A check meant to catch accidental renames missed cases where a model renamed a variable while making an otherwise identical edit. That got fixed and covered with new tests in one session, but the very next run hit the same bug anyway: it only worked when a parameter had a type annotation, and every new test written the day before happened to use parameters without one, just like the incumbent model always did. The tests were green, but they only proved it worked for one model.
The defect class moves up the stack
The first wave found contract blockers, runs that could not even start. The second wave found a run that completed cleanly, every gate green, and still shipped four quiet defects the harness caught before merge. That is the gates-absorb-the-work claim, measured rather than asserted.
Running it as a practice
A full wave, three models across three chains, costs about nineteen dollars and ninety minutes. The one hard rule is a code freeze during grading, because fixing a defect between arms turns your comparison into a comparison of two different pipelines.
What this does not prove
Every prompt here was tuned against the incumbent model, so when a new model breaks a stage, “latent pipeline defect” and “prompt does not fit this model” are both live explanations, and I usually cannot cleanly separate them. There is also a counter-lesson: on a separate bug, most of what looked like a capability ceiling on a cheap model turned out to be a pipeline deficit a stronger model would have quietly papered over. Model diversity is a fuzzer, not an argument for running the expensive tier in production. And the scope is narrow: one bug class, one repository, four models. These are existence proofs, not rates.
The part I would take to a different codebase is smaller than the tier comparison it came from: if a boundary in your system has only ever seen output from one generator, its contract is unproven no matter how green the suite is, and the cheapest way to prove it is to change the generator.
Runs described here are from a model-tier benchmark against pytest-xdist, a public open-source project, over 2026-07-19 to 2026-07-24. Four models across two waves plus their shakedown chains. Defect counts are filed rows in the project’s own gap catalogue, not a curated list. One bug class, one repository. Still R&D.