What this post is
In May I wrote about four architectural eras of the autonomous engineering pipeline I am building. Each era removed one more decision from the LLM and answered it from machine-extracted facts instead. Era 2 went as far as retiring the LLM Test Writer entirely, replacing it with a deterministic materializer that rendered structured test plans to code.
Three months and over a thousand runs later, the pipeline has been through four more eras, and the arc reversed direction. The Coder and the Test Writer are LLMs again, writing code and tests directly with edit tools inside a sandboxed copy of the tree. That reads like a retreat from everything the first four eras stood for. It is not, and explaining why is most of this post.
The first four eras were about taking decisions away from the model because nothing could check its work. The next four were about building checks strong enough that the model could have some of the work back.
The pipeline stopped trying to make the model incapable of being wrong and started making it impossible for wrong output to ship.
Those are different engineering programs, and the second one turned out to be the one that generalizes.
The pipeline now runs against TypeScript and Python codebases, including a ~200k line TypeScript monorepo, with a run archive around 1,550 runs. Still R&D.
Era 5: Certified authorship
The deterministic materializer was correct by construction and that was its ceiling. Every test structure it could render had to be anticipated by the structured plan schema, every assertion pattern needed a renderer, and each new ticket family meant extending the grammar before the pipeline could test it. After nineteen consecutive dead ends on one ticket family, where the plan schema kept being one expressiveness step behind what the test needed to say, the materializer arm lost to the alternative it was built to replace.
The alternative is an LLM with edit tools, working inside a sandboxed copy of the project. The Coder moved first. On the day it landed, the edit-tools Coder went three for three on a benchmark where the structured-operations path had gone zero for two. The Test Writer followed within days.
What makes this different from the pre-Era-1 pipeline, where an LLM also wrote everything, is what happens around the authorship:
The model edits a sandbox, never the tree. A derivation step diffs the sandbox against the pristine copy and converts the change into surgical operations: replace this function body, add this import, remove that one. Imports get peeled into their own structural operations rather than riding along inside a whole-file rewrite. If the diff cannot be expressed surgically, the fallback is a whole-file replacement guarded by a preservation check that compares every symbol the ticket did not touch against the original, byte for byte. A model that quietly deletes a function it found inconvenient gets caught by the differential, and a repeated refusal to preserve it routes to an adjudication step that decides whether the deletion was licensed by the ticket or is avoidance.
The tests the model writes still face the same two end-of-ticket gates the materializer’s output faced: confirm-RED, which proves the test fails when the change is absent, and the mutation gate, which proves it fails when the change is wrong. Authorship moved back to the model. The burden of proof did not move with it.
Era 2’s claim was that deterministic rendering eliminates authoring variance, and it did. The concession is that the variance was never the problem worth paying schema-maintenance for. Wrongness was, and the gates catch wrongness regardless of who authored the artifact. The other cost is plainer: an authoring model burns tokens a renderer never did, so a run costs more than it did in the materializer era. What that buys is reach. Ticket families that would each have demanded another schema extension before the pipeline could test them now need only a plan and the gates.
The materializer path also meant hand-building authorship capability, schema extension by schema extension, per ticket family and per project, in direct competition with what every new model generation already does out of the box. Models ship knowing how to drive their own tools now. The edit-tools interface gets better with every release without me lifting a finger; a renderer only gets better when I extend it. I am one person doing R&D, and the only sane allocation is to spend the engineering that no model release will do for me: the registry, the derivation, the gates. Nothing on that side of the line improves with the next model release. Everything on the model’s side does.
Era 6: Supply
Promoting the builders back to LLMs exposed a problem the materializer had been hiding: the pipeline had never been good at giving a builder the files it needs.
The discovery was embarrassing in the specific way pipeline bugs tend to be. A supply chain that derives which files a builder should see had been silently dead for months. A migration to tool-calling had removed the field it keyed on, nothing rewired the consumers, and every manifest since had shipped with an empty reading list. Unit tests stayed green the whole time because they fed the consumers well-formed fixtures the live pipeline no longer produced. The builders coped by searching, and an LLM that searches a large codebase does not fail loudly. It reads a few pages, decides it has seen enough, and writes something plausible against a contract it invented. One transcript has the Test Writer announcing “Now I have everything I need” before authoring a mock for a function signature it had never read.
The response was three mechanisms, all shipped across a few sessions:
First, supply became derived rather than hoped for. The registry walks imports from the subject under test, resolves them through barrel files to the modules where symbols actually live, and copies the result into the builder’s sandbox before the turn starts. Type declaration files, which no call-graph walk can reach, are supplied from the file’s own import statements.
Second, starvation became a hard failure. If a builder asks to read a file that exists in the project but was not supplied, the run fails naming the file, on the reasoning that an under-supplied builder does not stop, it guesses. The gate was calibrated against 1,461 archived run directories before arming: eight would have failed, zero false positives.
Third, breadth got replaced with an index. Measurement showed builders read almost none of what they were given, two of twenty-five supplied files in one run, none of thirty-seven in another, while one hunting session burned nineteen turns paging through a file looking for a thirty-line function whose location the registry knew all along. Pushing more files into the prompt was the wrong direction. The pipeline now writes a symbol index into the sandbox, a registry-derived catalog mapping every supplied file to its symbols and line ranges, and the builder pulls what it needs. The first measured effect was the hunting loop collapsing: nineteen turns to nine, input tokens down by nearly 70%, on the same ticket.
The framework contracts got the same treatment. Test-framework typing errors kept costing repair invocations because the builder could not see the framework’s declaration files, so the pipeline now chases type references through the framework’s declarations, through aliases and re-exports, and renders the relevant signatures into the builder’s context. The validation run for that change type-checked clean on the first pass with no repair invocation, the cheapest roll of that day.
Era 7: The Scout
Everything above assumes a ticket exists. Someone read the bug report, found where the fix lives, checked the fix is expressible there, and wrote the ticket. On real issues that someone was me, at about ten minutes of code reading per issue, which is a strange tax to pay in front of an autonomous pipeline.
The Scout is a pre-run analyst stage that takes a raw issue in the reporter’s own words and derives what the ticket writer used to: the locus, meaning which files and symbols the fix belongs to, and the feasibility, meaning whether the fix is expressible at that locus under the repo’s own constraints. One early target had a one-line fix that the type system of the surrounding code could not accept without a prerequisite change to a protocol, and a pipeline run launched without knowing that dies at the type gate after spending the whole chain. The Scout surfaces the prerequisite as a declared, operator-confirmed input instead, and the prerequisite travels through the pipeline as structured data with its declaring file force-included in the builder’s scope.
The Scout runs as a small ensemble with a bounded tool budget, grounded against the registry, and it feeds the pre-autonomous grounding chat rather than replacing it. The chat gained hazard questions from the Scout’s findings: shared surfaces near the edit set, seams where the issue’s phrasing and the code’s structure disagree. The operator still confirms direction. What changed is that the pipeline now starts from a symptom rather than from a hand-written specification, and the first acceptance runs took a raw issue to a committed, gate-certified implementation twice in a row.
Running a bare CLI agent on the same raw issues, with the same model underneath, produced implementations that missed some part of the written contract in about 80% of rolls. On the reuse axis it was starker: five bare rolls out of five reimplemented inline a helper the issue explicitly named, and none called it.
Era 8: Honest gates, and the second language
The last era is quieter and I think it is the one that will age best: the gates themselves came under the same scrutiny as the model.
A gate is a claim, and a claim nobody audits is a prompt rule with better branding.
The mutation gate claims that a surviving mutant means the test suite cannot detect a wrong implementation. That claim turned out to have a hole: mutant generation scoped candidate mutations by textual diff, so when a ticket wrapped an existing block in a try or a timeout race and re-indented everything inside it, every re-indented line read as changed code and the gate demanded the ticket’s test suite discriminate behavior the ticket never touched. Two chains died on mutants of pre-existing code, and the failure read as “the oracle is insufficient” when the truth was “the gate asked an illegitimate question.” The delta is now computed on the abstract syntax tree (AST) instead: parse the before and after, compare normalized token streams, and only lines whose structure actually changed become mutation sites. Re-indentation and formatter reflow stop being “changes” in any language, and the same mechanism works for Python, where whitespace is meaning and the textual workaround was unsafe by construction.
Gates also gained an offline replay harness. Every deterministic gate can be re-run against the stashed artifacts of past chains, so a change to gate logic gets its first execution against recorded history, with intended verdict flips declared ahead of time and any undeclared flip failing the change. Gate code stopped being the one part of the pipeline that got tested in production.
And because a builder under lint pressure has one evasion the sandbox cannot structurally prevent, writing a suppression directive instead of fixing the violation, there is now a guard that compares suppression comments in the final tree against the baseline. A newly introduced directive is a terminal failure with no repair pass and no escape hatch, on the reasoning that a repair step asked to remove a suppression is exactly the actor that added it. It has not fired yet. Its silence is the healthy state, and the day it fires will be worth writing about on its own.
The second language is the newest move. The pipeline’s polyglot discipline was a design stance for a long time, seams built for a language that never arrived. It is arriving now: the pipeline has produced fully green chains against a real Python project, and the current work is parity, walking the capability surface language by language and closing the places where the TypeScript arm has a mechanism and the Python arm has a stub. Some of those closures are small. Some, like teaching the mock-topology catalog to read pytest fixtures, or making the mutation delta language-neutral, landed in the last week. TypeScript is the proven arm. Python is the arm where every assumption gets re-tested, and the re-testing keeps finding real bugs on both sides.
The pattern, revised
The May post ended with a principle: wherever a decision is mechanically computable from machine-extracted facts, the pipeline must compute and apply it, not surface the facts and trust the LLM to apply the rule. That principle survived all four new eras untouched. What changed is where its boundary sits.
Authoring turned out to be on the model’s side of the line. Writing a test that expresses an intent is interpretation, and the materializer era was an attempt to mechanize interpretation, which is why its schema never stopped growing. Deciding whether that test proves anything is derivation, and derivation kept moving into the pipeline: reachability, coverage, necessity, sufficiency, preservation, conformance.
The model writes. The pipeline supplies and certifies. Supply, because a model that has to search will guess instead. Certify, because a model that cannot be checked will eventually be confidently wrong, and the only version of that you can live with is the one that cannot ship.
The pipeline runs inside Docker on real tickets against TypeScript and Python codebases, including a ~200k line TypeScript monorepo. The measured claims are drawn from the run archive: the edit-tools Coder’s 3/3 versus 0/2 benchmark ran 2026-07-29; the nineteen dead ends preceding the edit-tools Test Writer resolved 2026-08-01; the first fully green Python chain landed 2026-07-30; the bare-CLI baseline (7 rolls, ~80% contract miss) ran 2026-08-02 and its reuse regrade (5/5 inline duplication) was graded 2026-08-07; the starvation gate was calibrated against 1,461 archived run directories on 2026-08-07; the symbol-index comparison (19 turns to 9, input tokens down by nearly 70%, on the same ticket) was measured 2026-08-08; the raw-issue acceptance runs committed twice on 2026-08-04; the first reuse-family full completion landed 2026-08-07 and the bench ticket’s full completion, with the mutant survivors falling from four to zero after the strengthen pass, on 2026-08-09; the AST-scoped mutation delta and the Python parity work landed 2026-08-09 and 2026-08-10. Still R&D.