Our AI software factory shipped a change, and our own scoring system gave it a 94.
Then an engineer actually read the diff. The build was clean, the code was reasonable, the tests passed, and it had solved the wrong problem entirely. Not a subtle miss. The ticket asked for one thing; the factory confidently delivered another, and our judge stamped it near-perfect.
The bad build didn't scare me. Autonomous systems produce bad builds; that's priced in. What scared me was the confident wrong score. A factory that occasionally ships garbage is a known quantity: you review harder, you slow down, you catch it. A factory paired with a gullible judge is worse than no factory at all, because it manufactures false confidence at machine speed. Every green number becomes suspect. The whole point of the system, trusting work you didn't personally supervise, collapses.
Here's the thesis I've landed on after months of building this: everyone is building agents that do work. Almost nobody is building the system that proves the work is good. And I've come to believe that second system is the actual product. The agents are increasingly a commodity. The proof engine is the moat.
What the factory is
Quick context, then back to the interesting part. The software factory takes tickets written by PMs and produces pull requests, with no engineer driving. To be precise about what "AI" means here: we didn't train a model. The factory is a harness: orchestration, prompts, tooling, and guardrails wrapped around off-the-shelf models. The models are rented; the system around them is the engineering. It clarifies the ticket, plans the work, builds it, and opens a PR ready for human review. Tickets in, PRs out.
The rule that governs it is stolen from Deming and lightly vandalized: In God we trust; every factory run must bring data. Nothing enters the record unscored. The proof engine is the service that enforces this. Every single run is graded against the ticket's own acceptance criteria, with cost, runtime, and evidence attached. Not a sample. Not the runs someone remembered to flag. Every run, no exceptions.
That's the machine. Now, the part where the machine lied to us.
Hardening the judge
The false 94 got a full post-mortem, the same way a production outage would. What fooled the judge was depressingly mundane: a plausible-looking diff, a confident self-report from the build agent about what it had done, and, critically, no check on the premise. The judge evaluated how well the problem was solved without ever asking whether it was the right problem. Given a competent-looking solution and a confident narrative, it did what LLMs do with confident narratives. It believed them.
Everything we built in response falls out of that one sentence. Here's the pipeline that exists now:
Each piece of this is a lesson we paid for.
The premise gate. "Did it solve the right problem" now outranks "did it solve the problem well," structurally. Before any quality scoring happens, the judge must establish that the work actually addresses the ticket. A wrong-premise run cannot score high. Not "is penalized" — cannot, by construction. This sounds obvious in retrospect, and it is, which is exactly why it's worth writing down: the most dangerous failure mode of our judge was one nobody thought to defend against, because a human reviewer would never make it. A human skims the ticket first, involuntarily. An LLM judge does whatever the pipeline makes it do, and our pipeline handed it a solution and asked "is this good?" Of course it answered the question it was asked.
Claims are audited, not trusted. Factory runs narrate themselves: "I searched the codebase," "I checked existing implementations," "I verified against the org's patterns." The judge used to take these claims as evidence. Now it treats them as allegations. If a run says it did diligence, the judge checks whether the transcript actually contains that diligence. This one also has a scar behind it: a research run once claimed org-wide due diligence and, in reality, missed an existing internal solution that would have halved the cost of the work. The claim was fluent, specific, and false. Today, that exact category of omission is caught: the judge verifies what a run says it checked, not just what it says it built.
Judges need regression tests too. This is the one I'd tattoo on the team if I could. The false-94 run is frozen, evidence and all, as a permanent baseline. An offline replay harness re-scores it, along with a growing library of other adjudicated runs, on every change to the judge: every prompt tweak, every rubric edit, every model swap. If the false 94 ever creeps back toward 94, CI fails. We do not merge judge changes that un-learn our own post-mortems. Your evaluator will regress in exactly the ways your code does, for exactly the same reasons, and if you're not replaying its known failures against every change, you're just vibing with extra steps.
Determinism as policy. Same evidence in, same score out. Enforced, not hoped for. A judge that gives the same run an 81 on Tuesday and a 68 on Thursday isn't measuring the run; it's measuring the weather. Every downstream decision we make assumes scores are comparable across time, so nondeterminism isn't a quirk, it's data corruption. We treat it accordingly.
The through-line: treat your evaluator like production code. Version it. Test it. Give it its own post-mortems. It's the component whose failures are quietest and cost the most.
From scores to decisions
An individual score is trivia. "Run 4,317 got an 86" tells you almost nothing on its own. The leverage is entirely in the rollup layer that sits on top of every score.
The centerpiece is a capability matrix: every category of work the factory attempts, graded into one of three trust tiers: autonomous (ship it, humans skim), supervised (ship it, humans review properly), or engineer-required (the factory isn't ready for this; don't pretend otherwise). The beautiful thing about this matrix is that the expansion roadmap writes itself. You don't debate in a meeting whether the factory can handle migrations yet. You look at the cell. When a supervised category's scores stabilize high, it graduates. When it doesn't, it doesn't. Opinions leave the room; data stays.
Drift arrows sit on top of the same data. Because every run is scored, a regressing factory (a bad prompt change, a degraded model, a new failure mode) shows up as a downward arrow within a handful of runs, not a quarter later when someone notices the review queue smells wrong. And since every run carries its cost in dollars rather than tokens, the same rollup doubles as the unit economics of the factory, legible even to people who will never read a transcript.
Unglamorous lessons
A few things that took real debugging and produce zero conference talks:
- No verdict without evidence. We had a bug where an unscored run could "beat" a scored one in comparisons, with absence of data quietly outranking presence of data. The fix wasn't a cleverer default; it was refusing to rule. If the evidence isn't there, the answer is "no verdict," never a synthesized one.
- Classify the work before you judge it. A bug fix and a research spike are different jobs with different definitions of success. One judge with one rubric flattens that difference and produces scores that are precisely comparable and completely meaningless. Different work gets different judges.
- Lifecycle correctness is trust infrastructure. Ghost runs, lost scores, runs stuck in limbo states: these feel like janitorial bugs until you realize every rollup above them is silently poisoned. The trust map is only as honest as the plumbing beneath it.
- Build for the reader you have. Our execs read "Steady / Variable / Erratic." They do not read standard deviations, and pretending otherwise means your carefully computed statistics inform exactly nobody. Translating rigor into language is part of the rigor.
The dial
The factory ships code. The proof engine ships something more valuable: earned, quantified, regression-tested confidence. And confidence is the thing that lets you turn the autonomy dial up. Nobody expands an AI workforce's mandate on vibes; they expand it when the trust map says a category has earned it, when the drift arrows are flat, and when the cost per change makes sense in dollars.
That's the real product. Not the agent that does the work, but the system that proves the work is good.
In God we trust. Every factory run must bring data.