Swapping the LLM Is a Fuzzer for Your Own Code
Changing one config field, the model, surfaced more than twenty latent defects that months of green runs on the incumbent model never touched.
Autonomous Engineering Pipeline
An autonomous engineering pipeline in Python that takes a ticket from input to tested, reviewed code with no human in the execution loop. The architecture behaves more like a compiler for LLM behavior than a conventional agent framework: LLMs handle the parts that require interpretation, non-LLM code handles everything that can be computed. The pipeline runs inside Docker against TypeScript and Python codebases, including a ~100k line TypeScript monorepo. Multiple ticket types now execute end-to-end with zero LLM source-code authoring, running on codebase structure alone; the pipeline delegates only new logic to the Coder. Behavioral oracle tests are proven non-vacuous via two end-of-ticket gates: confirm-RED (necessity) and the mutation gate (sufficiency: a derived wrong-version mutant must be killed by the oracle before the ticket closes). Every run produces a full artifact chain: per-stage LLM reasoning traces, structured manifests, Coder operations, Reviewer verdicts, and per-stage cost and token data. The run archive spans ~1,300 runs from February 2026, almost all of it the same handful of ticket families re-run to shake out non-determinism. All posts document real runs, real failures, and the architectural decisions behind them. Still R&D.
Architecture principles
- Data Path Principle
- Every fact an LLM copies is a fact it can hallucinate. Trust hierarchy (top to bottom): human chat-authored design > machine-extracted data > model-generated constraints > model narrative. Facts and decisions are never mixed.
- Lego Instructions Principle
- Manifest quality determines outcome more than builder quality. Every piece in the box (Synthesizer populates all facts), unambiguous steps (single source of truth per field), no impossible assemblies (feasibility gate rejects unsatisfiable criteria before any builder runs).
- Brain layer / builders boundary
- Builders execute only. If a builder is searching, locating, or deciding, that is brain work leaking downstream. The manifest was incomplete. The fix is always to push the decision back upstream.
Pipeline stages
- Grounding chat
- Pre-autonomous stage. The operator confirms intent, exclusions, and design decisions in a structured exchange grounded against the registry. Output is a machine-readable handoff the downstream pipeline reads as the source of truth.
- Assembly stage
- The autonomous run from structured handoff to a committed diff ready for PR review, or a halt. Sequences the Synthesizer, Planner, Coder, Test Writer, Debugger router, and Reviewer, enforcing a retry budget and wall-clock cap per stage plus a global token cap on the run. No human involvement inside it: the operator sets intent upstream and reviews the PR downstream.
- Synthesizer
- Populates the manifest with file paths, symbols, signatures, and dependency resolutions from the registry, Tree-Sitter, and language server before any builder runs.
- Planner
- Reads the ticket and codebase, produces a structured manifest of what to build and which symbols to change.
- Coder
- Implements the code change within the constraints the manifest sets.
- Test Writer
- Deterministic materializer that renders structured test plans to code. Replaced an earlier LLM stage in Era 2. Generates behavioral oracle tests derived from manifest operations, proven non-vacuous via two end-of-ticket gates: confirm-RED (necessity) and the mutation gate (sufficiency).
- Debugger router
- The Debugger router classifies each failure and dispatches to the stage that owns the fix: code-wrong to the Coder, manifest/test/ticket-wrong to an operator correction surface, and four of six end-of-ticket gates to a fail-safe Coder repair. The only stage that can redirect the pipeline. Diagnoses without inheriting the Coder's reasoning.
- Reviewer
- Checks scope and correctness before the ticket closes. Split into Test Reviewer and Impl Reviewer in Era 4.
Start here
- From LLM Luck to Structurally Guaranteed: One Ticket Across Four Architectural Eras
The architectural overview: four eras, one ticket, cost per run dropping from $0.385 to $0.074.
- From LLM Author to LLM Reviewer: An AI Coding Agent Authors a Production Feature With Zero LLM Code Generation
How the pipeline's structural floor rose high enough to commit a multi-layer feature without any LLM source-code authoring. The current state of the system.
- What Calls This Function? Why AI Coding Agents Need a Language Server
Why the pipeline uses language-server context instead of loading whole files.
- What the Symbol Registry Stores, and How It Stays Fresh
The registry that keeps agent context precise across a changing codebase.
- Stop Asking the Model What the Code Already Knows
Why every field a Planner emits that the codebase already knows is a dice roll, and how machine extraction eliminates it.
- The Lego Instructions: An Architectural Principle for AI Coding Agents
Why manifest quality determines outcome more than builder quality: the principle behind the Synthesizer's role.
- Correct Code, Wrong File: How the Write Gate Contains Scope Creep
How write scope is constrained so the right code lands in the right place.
- Prompt Rules Are Advisory; Validators Are Binding
Why structural validation matters more than instructions alone.
- Raising the Floor under the Coder: Generalizing an AI Coding Agent across Ticket Types
How the pipeline went from one zero-Coder ticket type to six, and why four architectural moves did most of the work.
- Opus vs Sonnet: Deterministic Gates Beat Model Tier in Code Fixes
The clearest evidence that the gates, not the model, decide correctness: four Claude tiers hit four different failure modes on the same bug, and a different deterministic gate caught every one.
Go deeper
- Build for the Second Language You Don't Have Yet: Polyglot Discipline in a Code-Generation Pipeline
How polyglot discipline caught a latent fixture-scanner bug while TypeScript was still the only language, which seams already span languages, and why a seam is a hypothesis until a second language flows through it.
- The Registry Is Not a Fixed Input: Extend It, Don't Ask the Model
Three cases where a new ticket type forced a new registry extraction instead of a new prompt, and the failure mode of testing against fixtures that the live registry never produces.
- The False Green Baseline: When a Passing Test Suite Hides a Broken Type-Check
How pre-existing LSP noise made a correct rename fail three times, and why gating on the change delta rather than the absolute error count is the fix.
- Proving a generated test can fail: mutation testing as a sufficiency gate for an AI coding agent
How the automated mutation gate proves each behavioral oracle is sufficient: three mutator families, the no-diff equivalence check, and four vacuous passes that had to be debugged before the gate could be trusted.
- The Decision an AI Coding Agent Can't Make Alone: Operator-Grounded Intent Capture
The three rules for deciding which operator decisions the pipeline derives, which it asks for, and which it refuses to default: and why the answer must stay as structured data.
- The Behavioral Oracle: Testing What an AI Coding Agent's Route Suite Structurally Cannot See
How the pipeline derives a behavioral test below the flat-stub boundary from the same operation that writes the guard, commits it as a gate, and proves it non-vacuous with mutation testing.
- The Debugger Becomes a Router: Sending Each Failure to the Stage That Owns the Fix
How the Debugger router dispatches each failure type to the stage that owns the repair, and why half the original destinations were already gone when it shipped.
- Agent Pipeline Grounding Chat: From Free-Form Q&A to Typed Fields
How operator design decisions moved from free-form Q&A into typed handoff fields, and how the grounding chat gained a four-layer structural defense against LLM drift.
- Grounding an Autonomous Engineering Pipeline in Operator Design: The Pre-Autonomous Chat Stage
How operator intent becomes a structured handoff before autonomous execution starts.
- You Can't Diagnose an LLM Pipeline from Output Alone
Observability: why structured run traces are required to diagnose pipeline failures.
- No Stage Runs Forever: Retry Budgets and Escalation in an Agent Pipeline
Reliability limits, retry budgets, and escalation boundaries.
- Per-Field Hallucination Fixes Hit a Ceiling: 248 Runs on an AI Coding Agent
Empirical validation across 248 runs showing why patch fixes were not enough.
- Why a Warning Is Worse Than a Hard Stop
Why warning-and-continuing produces output that looks correct but cannot be caught by any downstream gate.
- Why the Debugger Never Inherits the Coder's Reasoning
How the Debugger's isolation from the Coder's reasoning is a design requirement, not a constraint.
Deterministic Gates Beat Model Tier: Opus vs Sonnet in Code Fixes
Four Claude model tiers fixed the same bug end to end. Correctness never moved; a different gate caught each model's own failure.
Build for the Second Language You Don't Have Yet: Polyglot Discipline in a Code-Generation Pipeline
The discipline of asking whether each change holds for a second language caught a latent fixture-scanner bug before any second language existed to trigger it.
The Registry Is Not a Fixed Input: Extend It, Don't Ask the Model
Three times a new ticket type forced a new registry fact. Each was an extraction, not a prompt, and each kept the derivation deterministic forever after.
The False Green Baseline: When a Passing Test Suite Hides a Broken Type-Check
How an AI coding agent's type-check gate misattributed pre-existing LSP errors to a correct rename, and why gating on the change delta is the answer.
Raising the Floor under the Coder: Generalizing an AI Coding Agent across Ticket Types
An AI coding agent went from one ticket type to six, with four architectural moves that raised the floor to deliver zero-Coder results on four types.
Proving a generated test can fail: mutation testing as a sufficiency gate for an AI coding agent
How mutation testing proves a generated behavioral oracle is sufficient, through three ticket types and five false passes that nearly went undetected.
The Decision an AI Coding Agent Can't Make Alone: Operator-Grounded Intent Capture
A deterministic resolver that passed every unit test then failed 40% of runs. Operator confirmation worked where a smarter algorithm did not.
The Behavioral Oracle: Testing What an AI Coding Agent's Route Suite Structurally Cannot See
The pipeline derives a behavioral test from the same operation that writes the guard, runs it below the flat-stub boundary, and proves it via mutation testing.
Error Messages as a Hypothesis Ladder: Five Techniques
Five techniques that turn a raw compiler error into a frequency-ordered hypothesis ladder, and why the pattern matters more for LLM debuggers than human ones.
The Debugger Becomes a Router: Sending Each Failure to the Stage That Owns the Fix
The Debugger in the autonomous engineering pipeline now routes each failure to the stage that owns the fix. Half the original destinations no longer exist.
Agent Pipeline Grounding Chat: From Free-Form Q&A to Typed Fields
Two open quality items are closed. Operator design decisions flow through typed handoff fields. The grounding chat has a structural defense against LLM drift.
From LLM Author to LLM Reviewer: An AI Coding Agent Authors a Production Feature With Zero LLM Code Generation
The pipeline completed a multi-layer TypeScript feature with zero LLM code generation: 6/6 ticket tests, 959/959 suite, $0.308 versus $0.681.
Grounding an Autonomous Engineering Pipeline in Operator Design: The Pre-Autonomous Chat Stage
A new pre-autonomous chat stage lets the operator ground design decisions in the registry before the pipeline runs, adding a new top to the trust hierarchy.
From LLM Luck to Structurally Guaranteed: One Ticket Across Four Architectural Eras
Seven pipeline runs, one ticket, four architectural eras. Per-test cost dropped from $0.385 to $0.074 by replacing LLM guesswork with structural derivation.
Who Reviews the Swarm? Why Probabilistic Verification Fails at Scale
Swarm parallelism is a throughput solution applied to a reliability problem. Probabilistic verification of probabilistic output does not converge.
Engineering Around LLM Non-Determinism: The Architectural Follow-Up to 248 Runs
What shipped in the three weeks after the 248-run hallucination ceiling: removing the LLM from computable decisions and validating everything else.
When the Pipeline Should Ask Instead of Guess
When a ticket has two equally-plausible interpretations, a deterministic stage stops the pipeline and asks before any Coder agent runs.
Vendor-Agnostic by Configuration: Per-Stage Model Setup in an LLM Coding Agent
Each stage in the pipeline runs against its own model and vendor config. How that design enables per-stage cost control, model swaps, and vendor flexibility.
You Can't Diagnose an LLM Pipeline from Output Alone
The run archive is where pass/fail becomes diagnostic: per-stage operation logs, reasoning traces, and a correlation token that spans every stage.
No Stage Runs Forever: Retry Budgets and Escalation in an Agent Pipeline
How per-stage retry budgets, wall-clock timeouts, and a global token cap keep any stage from running indefinitely, with the Debugger as the most complex case.
How a Binding Validator Is Wired: Synchronous Pre-Commit, Structured Errors, Retry Folding
The mechanics behind a binding validator: why synchronous pre-commit timing, structured rejections, and retry folding are each individually load-bearing.
Prompt Rules Are Advisory; Validators Are Binding
When the model has a strong prior, naming the failure mode in the prompt doesn't prevent it. Prompt rules are advisory; validators are binding.
What the Symbol Registry Stores, and How It Stays Fresh
The data model behind the symbol registry: per-symbol records, file-level hashes, call-graph edges, and the invalidation strategy that keeps it current.
How Filename Lookups Flood an AI Coding Agent's Context Window
Looking up symbols by filename instead of full path pulls every `index.ts` in the project into the agent's context. One line changed. 20 results down to 1.
The Lego Instructions: An Architectural Principle for AI Coding Agents
Three properties of a Lego instruction set, mapped to an AI coding pipeline: why manifest quality matters more than builder quality.
Per-Field Hallucination Fixes Hit a Ceiling: 248 Runs on an AI Coding Agent
Bernoulli model predicted 36% first-pass success across 248 pipeline runs. Measured: 21%. The gap explains why per-field hallucination fixes have a ceiling.
Stop Asking the Model What the Code Already Knows
Every field a Planner emits that the codebase already knows is a dice roll. Machine extraction replaces those dice rolls with deterministic lookups.
Why Architecture Gaps Need a Close Condition, Not a Backlog
Why tracking known architectural gaps with specific close conditions is more useful than a backlog, and what makes each entry work.
Fixture-First Development as an Early Warning System for AI Pipelines
Fixture-first development as an early warning system for AI pipelines: the first real-project run confirmed three known gaps instead of discovering new ones.
How claude -p Silently Inflates Your Pipeline Token Costs
Using claude -p in a pipeline? The model has bash access you never granted. Each tool call re-sends your full context. One sentence cuts token spend by 52%.
Silent Data Destruction: The Write Path Bug in Agentic Pipelines
The Coder added a new function to an existing file. The pipeline reported success. All seven existing functions were gone.
Four Pipeline Bugs That Only Surface With Less Capable Models
A ticket that passed twice failed four times at lower model effort, exposing four structural pipeline bugs the higher-effort run had masked.
LLM Non-Determinism Is a Pipeline Failure, Not a Model Problem
Same ticket, same pipeline config, different result two days apart. Why the first run passing was not confirmation that the constraint was enforced.
Intentional Technical Debt: Building Features in the Wrong Order
The pipeline committed code before branch isolation existed. The risk was real, named, given a close condition. That is what makes it different from a shortcut.
Why a Warning Is Worse Than a Hard Stop
When the pipeline detects zero test files, logging a warning and continuing produces output that looks correct but cannot be caught by any downstream gate.
Correct Code, Wrong File: How the Write Gate Contains Scope Creep
On attempt 3, the Coder tried to write a file that was not in the manifest. The write gate stopped it before anything hit disk. This is what it is for.
Why the Debugger Never Inherits the Coder's Reasoning
The Debugger receives the test failure and the code on disk, not the Coder's reasoning. That isolation is not a constraint. It is the design.
What Calls This Function? Why AI Coding Agents Need a Language Server
Tree-Sitter tells you where a symbol is defined. It cannot tell you where it is called. That gap cost one pipeline run 33,000 tokens to find out.
The Quality Gate That Passed When It Failed
A Haiku optimization made the L2 quality gate silently pass on every run. The fix was removing the LLM call entirely.
Why AI Coding Agents Fail on Evolving Codebases
Not a model capability problem. An agent with the wrong codebase version produces output that is plausible but wrong in ways that are hard to catch.