Every measurement this pipeline has produced started with me finding the work: reading a real project’s issue tracker until something worth building turned up, or writing the ticket myself on the fixture. I do that part by hand, and it is the one step in an otherwise autonomous system that does not scale.
Since mid-August the work comes from LoopsBench instead, Microsoft’s long-horizon coding benchmark.
Why hand-mined benchmark supply does not scale
Finding one ticket worth running takes hours of reading. Most of what I open does not fit what I need to measure, and the search starts from zero on the next repository. A hand-mined corpus also samples one person’s sense of what is worth building, and nothing checks that against the work a project’s own maintainers would have prioritised. Before August there were three targets: a public Python repository, a TypeScript fixture I wrote myself, and a ~200k line TypeScript monorepo. All picked by me, one written by me, which is a thin base for claiming a pipeline generalises.
The maintainer cost of using someone’s tracker as a testbed
Reviewing a change you did not write means working out what the author meant and checking it against code you may not have opened in a year. Maintainers absorb that on their own time, while drive-by pull requests arrive from people who pointed a model at an issue tracker. Nothing this pipeline built has been sent upstream and publishing stays deferred, so none of that load has reached anyone. I think what this pipeline writes is better than that, and it makes no difference: a maintainer still has to read it, and nobody asked them to.
A benchmark task has already consented to being attempted. A stranger’s issue tracker never agreed to anything.
Why this benchmark and not SWE-bench
SWE-bench asks one question: does this patch make the tests pass? A lucky patch and a certified one score the same. Every gate this pipeline runs, confirm-RED and the mutation gate, the obligations carried between sub-tickets, none of it changes the score.
LoopsBench scores what those gates are for. A task is a set of units built in dependency order, most of them carrying their own tests. Once a unit passes, its tests keep running for the rest of the task, and the benchmark counts every one a later unit breaks. Not breaking earlier work is what this pipeline is built to guarantee, and here that is what gets measured.
Units that carry tests ship them hidden, and the pipeline never sees them, so the verdict is no longer the pipeline’s.
A system that grades its own work can only report its own standards back to itself.
How a benchmark task becomes a pipeline target
Every LoopsBench task ships its own Dockerfile and compose file, and the harness is public, so provisioning I used to do by hand was already done 112 times over. One command extracts a task’s starting tree, builds its image, lists the units whose prerequisites are met, and exports the patch series. A deterministic step renders one ticket per unit from the requirement prose. The locus and the numbered acceptance criteria still come from the grounding chat, and several downstream checks do nothing without them. Generation refuses a null intent, so no run skips that stage.
A copy of the working tree goes to a throwaway container that mounts the unit’s hidden test read-only and runs the benchmark’s command. The reference patch ships inside the task image, outside the working directory. The pipeline confines itself to the target path and never reads it.
One Python unit against pytest and one TypeScript unit against Jest have gone end to end, and the referee accepted both.
Each unit also gets a grade the benchmark does not ask for. A unit’s gold patch is the change it was built from; the grade says how close the pipeline came to it. Gold-identical is the same edit, gold-placed is different code in the same place, gold-equivalent a change that works somewhere else.
Both units landed in gold-equivalent. The gold patch applies the constraint once, at the producer. The pipeline applied it at every site that consumes the value, in the right file, one step downstream. A fail-to-pass score cannot see that difference.
What this does not prove yet
The tasks come from public repositories, so a model may have seen the real fix during training. That was true of the previous public target too. The unit requirements were written by a model and checked against the benchmark authors’ standard, cleaner than any real tracker’s, so any triage number here is a best case. Resolved means the referee accepted the change. It does not mean a maintainer would have written it that way.
Two units out of more than 5,300 is an existence proof, not a rate. No run has walked a task’s full dependency graph yet, so there is no resolve rate or regressions-per-run figure to compare with the baselines. Building the component that walks a task’s ready frontier is the next step.
Benchmark: LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation, Han Li et al., Microsoft and collaborators (arXiv:2608.00267v2). Project site: loopsbench.ai. Harness and tasks: microsoft/Loopsbench (MIT). Runs use dataset 84ae415, harness commit 5880b64. Task data is canaried against training corpora and is not reproduced here.