Green Tests, Red-Handed: Catching an AI Coding Agent Silencing the Type Checker

On a pinned pytest-xdist checkout, a run produced everything a ticket is judged on. The generated test file was green at 36 passed, the pipeline’s type-check gate reported zero errors, and the change did what the ticket asked. The pipeline refused to ship it. The verdict came from a guard with one job: catch any suppression directive an AI coding agent introduced into code it was writing, # type: ignore, # noqa, eslint-disable, and the rest of that family.

The ticket derives from pytest-xdist issue #880: run pytest with --lf and -n together and the run-last-failure: summary line goes missing. The Coder’s job is to re-emit that line from xdist’s controller, and the code that does it reads the pytest cache. The working version of the change arrived with one new comment on that read: # type: ignore[union-attr]. The guard rejected it and told the Coder to remove the directive and fix the underlying problem. The Coder rewrote the change, from a fresh plan, and reached for the same comment. The run ended refused, with the tree reset.

Why one comment gets its own guard

A suppression comment is an evasion the sandbox cannot stop. Every cruder way to beat a lint gate requires touching something outside the files the builder was given, and those writes are already refused. A comment is legal syntax in a file the builder is supposed to work on. It also defeats the checks on both sides of the run, because both ask the same checker and the checker honors the directive.

So the guard compares each file the builder touched against the state it started from. It fires only on a suppression that was not already there. A directive that merely moved lines is not a new one, and a directive quoted inside a string is not a directive. The guard runs twice: once while the Coder is still working, where a finding comes back as feedback, and once as an independent end-of-ticket check, where it ends the run. As it stands there is no repair pass and no override. The likely outcome of a repair pass is a model politely re-adding the comment.

The rule ignores motive on purpose. The comment hides nothing about the change itself: the diff shows the edit, the tests run, both Reviewers read the code. What it silences is the checker, on this run and on every edit that line receives afterwards.

A suppression directive is not a comment about today’s code. It is a permanent instruction to the checker, inherited by every future edit to that line, written by an agent that will not be around to defend it.

The suppressed error does not exist

I audited the incident expecting to find an unguarded Optional under the directive. Here is what the Coder wrote in dsession.py:

lastfailed: dict[str, bool] = self.config.cache.get(  # type: ignore[union-attr]
    "cache/lastfailed", {}
)

pytest annotates the attribute as plain Cache, and its own cache plugin reads it without one:

class Config:
    # Set by cacheprovider plugin.
    cache: Cache

# _pytest/cacheprovider.py
self.lastfailed: dict[str, bool] = config.cache.get("cache/lastfailed", {})

Both of those lines sit in source the pipeline had already rendered into the model’s context. No checker on this project ever raises the complaint the directive pre-empts. I think the model imported a defensive habit from its training data, an older annotation of the same API, and it held onto the comment even after the guard told it to remove it.

The target’s own tooling said so at the time, mechanically. xdist runs mypy in strict mode, strict mode warns on unused ignores, and the project conventions check failed whenever the directive was present, with error: Unused "type: ignore" comment. The checker certified that the directive suppresses nothing. On this project the patch was caught twice over: the same check runs in xdist’s own CI, so this particular ship would have failed there even without the guard. That warning only catches this variant: an ignore that hides a real error is the case an unused-ignore warning cannot flag, and that warning is off by default until a project opts into strict mode. The guard does not care which variant it is looking at.

Run the same ticket again and the coin lands the other way

The comment is not a fixed property of the ticket. The re-roll used the same ticket, the same model, and the same configuration, and it wrote the same cache read with no directive at all. Every gate came back clean, the project’s own suite ran 224 tests and passed every one, and the ticket shipped. One draw in two reaches for a defensive suppression unprompted. The guard is not policing a deterministic defect, it is policing a coin flip. The losing side of that flip lands a directive with no error under it in most real codebases, with every test green.

Ninety-one silent runs, then one firing

The guard was built before anyone had seen the behavior it catches. The reasoning was that if a builder ever papers over its own mistake, every other signal will be green and nothing else will catch it. Across the 91 runs archived between arming and the incident it examined every Coder turn and every end-of-ticket tree and said nothing. Then it spoke on one run, correctly.

A guard that never fires produces no evidence that it works. The only measurement it can offer is the run it refuses.

A comment scan has a boundary, and writing up the firing found it: a type checker can also be silenced by ordinary code, cast(...), an Any annotation, as any in TypeScript, none of which leaves a comment to find. The same review armed a counter for that channel. A test suite measures behavior, and the suppressed patch behaved perfectly. What the guard measures is the one thing the suite has no opinion about: what the code told its checkers to stop looking at in order to get there.

The suppression guard described here has one live incident, a true positive, and zero false positives across 91 archived runs; the counter armed for casts and Any annotations has never fired. One incident and one re-roll is a coin flip’s worth of evidence about the behavior’s rate, not a benchmark. Still R&D.