Changing Where the Work Comes From Exposed 145 Defects in an AI Coding Pipeline
The first benchmark unit reported completion in 114 seconds, every gate green. Its hidden test refuted it. Twelve days of external targets filed 145 defects.
Autonomous Engineering Pipeline
An autonomous engineering pipeline in Python that takes a ticket from input to tested, reviewed code with no human in the execution loop. The architecture behaves more like a compiler for LLM behavior than a conventional agent framework: LLMs handle the parts that require interpretation, non-LLM code handles everything that can be computed.
The pipeline runs inside Docker. It is proven on TypeScript and Python codebases, including a ~200k line TypeScript monorepo, both of which are now held as regression corpora; current work comes from LoopsBench, Microsoft's published long-horizon coding benchmark, whose hidden tests grade each result from outside the pipeline. The pipeline delegates only new logic to the Coder. The Coder and Test Writer author with edit tools in sandboxed copies of the tree; the pipeline supplies their context from the registry and certifies their output rather than rendering it deterministically.
Behavioral oracles are proven non-vacuous via two gates: confirm-RED (necessity, before the implementation exists) and the end-of-ticket mutation gate (sufficiency: a derived wrong-version mutant must be killed by the oracle before the ticket closes).
Every run produces a full artifact chain: per-stage LLM reasoning traces, structured manifests, Coder operations, Reviewer verdicts, and per-stage cost and token data. The run archive spans ~1,900 runs from April 2026, almost all of it the same handful of ticket families re-run to shake out non-determinism. All posts document real runs, real failures, and the architectural decisions behind them. Still R&D.
Architecture principles
- Data Path Principle
- Every fact an LLM copies is a fact it can hallucinate. Trust hierarchy (top to bottom): human chat-authored design > machine-extracted data > model-generated constraints > model narrative. Facts and decisions are never mixed.
- Lego Instructions Principle
- Manifest quality determines outcome more than builder quality. Every piece in the box (Synthesizer populates all facts), unambiguous steps (single source of truth per field), no impossible assemblies (feasibility gate rejects unsatisfiable criteria before any builder runs).
- Brain layer / builders boundary
- Builders execute only. If a builder is searching, locating, or deciding, that is brain work leaking downstream. The manifest was incomplete. The fix is always to push the decision back upstream.
Pipeline stages
- Scout
- Analyst stage inside the grounding chat. Takes the problem prose as written and derives the fix's locus and feasibility, including prerequisites the type system demands, before any autonomous run starts. Ensemble with a bounded tool budget, grounded against the registry.
- Grounding chat
- Pre-autonomous stage. The operator confirms intent, exclusions, and design decisions in a structured exchange grounded against the registry. Output is a machine-readable handoff the downstream pipeline reads as the source of truth. Scout findings feed it on raw-issue runs.
- Assembly stage
- The autonomous run from structured handoff to a committed diff ready for review, or a halt. Sequences the Synthesizer, Planner, Coder, Test Writer, Debugger router, and Reviewer under per-stage retry and wall-clock caps plus a global token cap. The sub-ticket attempt loop runs the full ticket oracle at final acceptance, so a chain fails inside the loop rather than at the end of the ticket. End-of-ticket gates certify the diff: type-check, lint and conventions, the mutation gate, the full target suite, and a suppression guard. On benchmark work, tests the pipeline never sees grade the result out of band, and each completed unit is additionally graded for how close its change is to the gold patch that ships with the task. No human involvement inside it.
- Synthesizer
- Compiles planner-authored sub-tickets into builder manifests with no LLM in the conversion, grounded against a Tree-Sitter and language server derived symbol registry. Builder context is derived separately: supplied files, a symbol index over them, framework contracts, and registry-derived contract signatures pushed into the prompt for every symbol the manifest references.
- Planner
- Decomposes the ticket into sub-tickets naming what to build and which symbols to change. Authors decisions, not facts: the Synthesizer compiles them into builder manifests. Receives re-plans on a manifest-wrong diagnosis.
- Coder
- LLM with edit tools in a sandboxed copy of the tree. Supplied up front with registry-derived files, a symbol index, and pushed contract signatures for referenced symbols; hard-fails if a needed real file was not supplied, except files withheld by design. Sandbox diffs are converted to surgical operations; a rewrite that is not symbol-decomposable is refused until the final attempt, which may fall back to a whole-file rewrite guarded by a byte-level preservation differential on untouched symbols.
- Test Writer
- LLM test author with edit tools that reads the target tree in place and writes only into a scratch directory, with a registry-derived symbol index, pushed contract signatures, and framework type contracts. Oracles are proven non-vacuous via confirm-RED (necessity) and the mutation gate (sufficiency), with mutants scoped to the ticket's semantic delta on the abstract syntax tree.
- Debugger router
- The Debugger router classifies each failure and dispatches to the stage that owns the fix: code-wrong to the Coder, manifest-wrong to the Planner for a re-plan, machine-confirmed ticket contradictions to an operator correction surface, test-wrong to a bounded automated test-repair pass, and most end-of-ticket gates to a fail-safe Coder repair. The only stage that can redirect the pipeline. Diagnoses without inheriting the Coder's reasoning.
- Reviewer
- Checks scope and correctness before the ticket closes. Split into Test Reviewer and Impl Reviewer in Era 4.
Start here
- Four More Architectural Eras: Giving Authorship Back to an AI Coding Agent
History through Era 8: authorship returns to the model under certification. Edit-tools builders, derived supply, the pre-run Scout, and gates honest enough to be audited themselves.
- From LLM Luck to Structurally Guaranteed: One Ticket Across Four Architectural Eras
How the architecture got here: four eras, one ticket, replacing LLM guesswork with structural derivation step by step. History through Era 4. The follow-up eras, where authorship returns to the model under certification, are covered by Four More Architectural Eras.
- Where an Autonomous Engineering Pipeline Gets Its Work: LoopsBench
Where the pipeline's work now comes from, and why an external referee holding tests the pipeline never sees replaced a human mining issue trackers for something worth building.
- What Calls This Function? Why AI Coding Agents Need a Language Server
Why the pipeline uses language-server context instead of loading whole files.
- What the Symbol Registry Stores, and How It Stays Fresh
The registry that keeps agent context precise across a changing codebase.
- Stop Asking the Model What the Code Already Knows
Why every field a Planner emits that the codebase already knows is a dice roll, and how machine extraction eliminates it.
- The Lego Instructions: An Architectural Principle for AI Coding Agents
Why manifest quality determines outcome more than builder quality: the principle behind the Synthesizer's role.
Where an Autonomous Engineering Pipeline Gets Its Work: LoopsBench
Hand-mining GitHub issues was the one step in an autonomous engineering pipeline that did not scale. LoopsBench replaced it with 112 tasks and a referee.
Green Tests, Red-Handed: Catching an AI Coding Agent Silencing the Type Checker
An AI coding agent got every test green and carried a # type: ignore for an error that does not exist. A guard silent for 91 runs refused the ship.
Four More Architectural Eras: Giving Authorship Back to an AI Coding Agent
The first four eras took decisions away from the LLM. The next four gave authorship back, once the gates were strong enough to certify what it wrote.
Deletion Avoidance in AI Coding Agents Is Duplicate Code in a Mirror
Frontier models avoid deleting code. The same question asked the other way produces duplicates: five of five unassisted runs added a parallel copy.
Swapping the LLM Is a Fuzzer for Your Own Code
Changing one config field, the model, surfaced more than twenty latent defects that months of green runs on the incumbent model never touched.
Deterministic Gates Beat Model Tier: Opus vs Sonnet in Code Fixes
Four Claude model tiers fixed the same bug end to end. Correctness never moved; a different gate caught each model's own failure.
Build for the Second Language You Don't Have Yet: Polyglot Discipline in a Code-Generation Pipeline
The discipline of asking whether each change holds for a second language caught a latent fixture-scanner bug before any second language existed to trigger it.
The Registry Is Not a Fixed Input: Extend It, Don't Ask the Model
Three times a new ticket type forced a new registry fact. Each was an extraction, not a prompt, and each kept the derivation deterministic forever after.
The False Green Baseline: When a Passing Test Suite Hides a Broken Type-Check
How an AI coding agent's type-check gate misattributed pre-existing LSP errors to a correct rename, and why gating on the change delta is the answer.
Raising the Floor under the Coder: Generalizing an AI Coding Agent across Ticket Types
An AI coding agent went from one ticket type to six, with four architectural moves that raised the floor to deliver zero-Coder results on four types.
Proving a generated test can fail: mutation testing as a sufficiency gate for an AI coding agent
How mutation testing proves a generated behavioral oracle is sufficient, through three ticket types and five false passes that nearly went undetected.
The Decision an AI Coding Agent Can't Make Alone: Operator-Grounded Intent Capture
A deterministic resolver that passed every unit test then failed 40% of runs. Operator confirmation worked where a smarter algorithm did not.
The Behavioral Oracle: Testing What an AI Coding Agent's Route Suite Structurally Cannot See
The pipeline derives a behavioral test from the same operation that writes the guard, runs it below the flat-stub boundary, and proves it via mutation testing.
Error Messages as a Hypothesis Ladder: Five Techniques
Five techniques that turn a raw compiler error into a frequency-ordered hypothesis ladder, and why the pattern matters more for LLM debuggers than human ones.
The Debugger Becomes a Router: Sending Each Failure to the Stage That Owns the Fix
The Debugger in the autonomous engineering pipeline now routes each failure to the stage that owns the fix. Half the original destinations no longer exist.
Agent Pipeline Grounding Chat: From Free-Form Q&A to Typed Fields
Two open quality items are closed. Operator design decisions flow through typed handoff fields. The grounding chat has a structural defense against LLM drift.
From LLM Author to LLM Reviewer: An AI Coding Agent Authors a Production Feature With Zero LLM Code Generation
The pipeline completed a multi-layer TypeScript feature with zero LLM code generation: 6/6 ticket tests, 959/959 suite, $0.308 versus $0.681.
Grounding an Autonomous Engineering Pipeline in Operator Design: The Pre-Autonomous Chat Stage
A new pre-autonomous chat stage lets the operator ground design decisions in the registry before the pipeline runs, adding a new top to the trust hierarchy.
From LLM Luck to Structurally Guaranteed: One Ticket Across Four Architectural Eras
Seven pipeline runs, one ticket, four architectural eras. Per-test cost dropped from $0.385 to $0.074 by replacing LLM guesswork with structural derivation.
Who Reviews the Swarm? Why Probabilistic Verification Fails at Scale
Swarm parallelism is a throughput solution applied to a reliability problem. Probabilistic verification of probabilistic output does not converge.
Engineering Around LLM Non-Determinism: The Architectural Follow-Up to 248 Runs
What shipped in the three weeks after the 248-run hallucination ceiling: removing the LLM from computable decisions and validating everything else.
When the Pipeline Should Ask Instead of Guess
When a ticket has two equally-plausible interpretations, a deterministic stage stops the pipeline and asks before any Coder agent runs.
Vendor-Agnostic by Configuration: Per-Stage Model Setup in an LLM Coding Agent
Each stage in the pipeline runs against its own model and vendor config. How that design enables per-stage cost control, model swaps, and vendor flexibility.
You Can't Diagnose an LLM Pipeline from Output Alone
The run archive is where pass/fail becomes diagnostic: per-stage operation logs, reasoning traces, and a correlation token that spans every stage.
No Stage Runs Forever: Retry Budgets and Escalation in an Agent Pipeline
How per-stage retry budgets, wall-clock timeouts, and a global token cap keep any stage from running indefinitely, with the Debugger as the most complex case.
How a Binding Validator Is Wired: Synchronous Pre-Commit, Structured Errors, Retry Folding
The mechanics behind a binding validator: why synchronous pre-commit timing, structured rejections, and retry folding are each individually load-bearing.
Prompt Rules Are Advisory; Validators Are Binding
When the model has a strong prior, naming the failure mode in the prompt doesn't prevent it. Prompt rules are advisory; validators are binding.
What the Symbol Registry Stores, and How It Stays Fresh
The data model behind the symbol registry: per-symbol records, file-level hashes, call-graph edges, and the invalidation strategy that keeps it current.
How Filename Lookups Flood an AI Coding Agent's Context Window
Looking up symbols by filename instead of full path pulls every `index.ts` in the project into the agent's context. One line changed. 20 results down to 1.
The Lego Instructions: An Architectural Principle for AI Coding Agents
Three properties of a Lego instruction set, mapped to an AI coding pipeline: why manifest quality matters more than builder quality.
Per-Field Hallucination Fixes Hit a Ceiling: 248 Runs on an AI Coding Agent
Bernoulli model predicted 36% first-pass success across 248 pipeline runs. Measured: 21%. The gap explains why per-field hallucination fixes have a ceiling.
Stop Asking the Model What the Code Already Knows
Every field a Planner emits that the codebase already knows is a dice roll. Machine extraction replaces those dice rolls with deterministic lookups.
Why Architecture Gaps Need a Close Condition, Not a Backlog
Why tracking known architectural gaps with specific close conditions is more useful than a backlog, and what makes each entry work.
Fixture-First Development as an Early Warning System for AI Pipelines
Fixture-first development as an early warning system for AI pipelines: the first real-project run confirmed three known gaps instead of discovering new ones.
How claude -p Silently Inflates Your Pipeline Token Costs
Using claude -p in a pipeline? The model has bash access you never granted. Each tool call re-sends your full context. One sentence cuts token spend by 52%.
Silent Data Destruction: The Write Path Bug in Agentic Pipelines
The Coder added a new function to an existing file. The pipeline reported success. All seven existing functions were gone.
Four Pipeline Bugs That Only Surface With Less Capable Models
A ticket that passed twice failed four times at lower model effort, exposing four structural pipeline bugs the higher-effort run had masked.
LLM Non-Determinism Is a Pipeline Failure, Not a Model Problem
Same ticket, same pipeline config, different result two days apart. Why the first run passing was not confirmation that the constraint was enforced.
Intentional Technical Debt: Building Features in the Wrong Order
The pipeline committed code before branch isolation existed. The risk was real, named, given a close condition. That is what makes it different from a shortcut.
Why a Warning Is Worse Than a Hard Stop
When the pipeline detects zero test files, logging a warning and continuing produces output that looks correct but cannot be caught by any downstream gate.
Correct Code, Wrong File: How the Write Gate Contains Scope Creep
On attempt 3, the Coder tried to write a file that was not in the manifest. The write gate stopped it before anything hit disk. This is what it is for.
Why the Debugger Never Inherits the Coder's Reasoning
The Debugger receives the test failure and the code on disk, not the Coder's reasoning. That isolation is not a constraint. It is the design.
What Calls This Function? Why AI Coding Agents Need a Language Server
Tree-Sitter tells you where a symbol is defined. It cannot tell you where it is called. That gap cost one pipeline run 33,000 tokens to find out.
The Quality Gate That Passed When It Failed
A Haiku optimization made the L2 quality gate silently pass on every run. The fix was removing the LLM call entirely.
Why AI Coding Agents Fail on Evolving Codebases
Not a model capability problem. An agent with the wrong codebase version produces output that is plausible but wrong in ways that are hard to catch.