Testing

Changing one config field, the model, surfaced more than twenty latent defects that months of green runs on the incumbent model never touched.

Four Claude model tiers fixed the same bug end to end. Correctness never moved; a different gate caught each model's own failure.

How an AI coding agent's type-check gate misattributed pre-existing LSP errors to a correct rename, and why gating on the change delta is the answer.

How mutation testing proves a generated behavioral oracle is sufficient, through three ticket types and five false passes that nearly went undetected.

The pipeline derives a behavioral test from the same operation that writes the guard, runs it below the flat-stub boundary, and proves it via mutation testing.

Fixture-first development as an early warning system for AI pipelines: the first real-project run confirmed three known gaps instead of discovering new ones.

A ticket that passed twice failed four times at lower model effort, exposing four structural pipeline bugs the higher-effort run had masked.

Same ticket, same pipeline config, different result two days apart. Why the first run passing was not confirmation that the constraint was enforced.

When the pipeline detects zero test files, logging a warning and continuing produces output that looks correct but cannot be caught by any downstream gate.

The Debugger receives the test failure and the code on disk, not the Coder's reasoning. That isolation is not a constraint. It is the design.

A Haiku optimization made the L2 quality gate silently pass on every run. The fix was removing the LLM call entirely.