great_cto runs Claude Code as a pipeline of 70 specialist agents. Each agent has an eval file: hand-written cases, a split into tuning and holdout, an LLM actor that plays the agent from its real prompt, and a judge model that grades each answer against a pass criterion.
I scored all 70 agents out of 100 and got a list of offenders. cli-reviewer at 0.12. code-reviewer at 0.22. msp-reviewer at 0.22. The plan was obvious: read the failures, rewrite the prompts, re-run.
Before rewriting anything I read the failed cases. By the end of the day, seventeen agents had been re-measured, all seventeen passed, and one prompt had changed.
1. The judge graded answers that were not there
cli-reviewer's last run had eight cases. Seven recorded answers were empty strings. The judge received each one, dutifully wrote "the response is empty and does not name argv arrays", and marked it FAIL. The score, 0.12, was a measurement of nothing the agent did.
The runner already refused to trust an empty judge reply. It had no such rule for the actor. Now an empty, refused, or cut-to-nothing answer is recorded as a case that did not happen, counted as dropout, and never sent to the judge. Re-run on the current runner: 8/8.
2. A gate that counted someone else's pass
The coverage ladder answers "how much do we actually know about this agent": missing, present, exercised, passing. It decided architect was passing.
architect's own evals were failing. The pass came from EVAL-ai-prompt-architect-versioning — a different agent's eval, matched because its file name contains the string "architect". A pack of voice evals did the same for voice-ai-reviewer: they are run with a generic actor that plays the voice agent on a live call, and their passes were being credited to the reviewer.
A run now counts toward an agent only when that agent was the actor.
3. A 96% run recorded as failed
After a fix below, code-reviewer scored 24 of 25. The runner printed a pass. The history file said failed: even the high end (0.99) is below 1.00.
The eval has two bars: 5/5 on tuning, 2/3 on holdout. The pass/fail line judged each split against its own bar. The confidence interval took the leading bar, 1.0, and applied it to all 25 cases — and the status page that reads history compared 0.96 to 1.0 and put a passing reviewer on the failing rung. Now both splits get their own interval.
4. The cases withheld what the agent is required to ask for
This was most of it.
code-reviewer's cases described changes in prose: "Diff adds a DB call inside a .map()." The agent's prompt requires every finding to cite file:line with evidence. There was no file and no line. It answered, correctly, that it had nothing to cite. Fourteen of 25.
I wrote a candidate prompt telling it to review what it was given anyway. 13 of 25 — and it started blocking cases that had passed before. I did not ship it. Instead the cases got a file and a line, and the holdout cases got one sentence saying they are replies during a review of a diff already read. Expected answers unchanged. Same prompt: 24 of 25.
senior-dev failed a case asking it to write a failing test first. Its prompt works from an IMPL-BRIEF, and without one falls back to the architecture doc. The case had neither, so it asked what it was allowed to touch. With a brief in the scenario: 5/5 across three samples.
5. The eval contradicted the prompt
product-owner scored 19 of 29. Nearly every failure was the same: given an idea with missing facts, it asked questions and stopped instead of writing a brief.
Its prompt says exactly that, since a change in August: if who has the problem, what it costs them, why now, or what success looks like is missing — ask, and stop. The eval's scenario, written in June, said the opposite: carry a default and keep going. The agent was being graded against a rule it had been told to break.
That is not a prompt bug or an eval bug. It is a product decision, and it went to a human. The decision was ask-and-stop; the eval now says so and supplies the four facts to every case that is not about them.
6. The one real miss
With the ruler straight, one failure repeated. Given an idea that is a smaller copy of an incumbent with no wedge anyone can name, product-owner invented a wedge and recommended PIVOT with a full brief — in two runs out of three. The prompt required naming an incumbent but said nothing about what to do when there is no wedge, so the section got filled.
One paragraph: no wedge found is a finding, and the decisive reason for DON'T BUILD. Tuning went to 30/30 over three samples; holdout 14/20 against 15/20 before, inside noise.
7. The number I got wrong
In the morning I reported that 41 of the 123 failing cases had an empty answer. It went into a commit message.
It was wrong. Older history rows predate the field that stores the answer, and my count read "no answer on record" as "empty answer". It also counted passing cases in a total I described as failing. Measured properly: 7 recorded empty answers, 10 cases with no answer on record, 105 with real text. The code change stood; the headline did not. The commit is public and stays as it is; the correction is a separate commit the same day.
It is the same bug as the rest of this post, one level up: a number derived from a record, trusted without checking what the record could actually say.
What I would tell anyone running LLM evals
- Read the failed answers before touching the prompt. Every change above started from the answer text, not the score.
- A case must contain what the agent is required to ask for. Otherwise a disciplined agent fails for being disciplined.
- An eval can disagree with the prompt. When it does, a person decides which one is the spec.
- Candidate prompts get A/B'd, and losing candidates do not ship.
- Record the answer. The fixes above were possible only because the answer text was on record; the ones I could not check were the rows without it.
The runner, the eval files and the ladder are in the repository: tests/eval/. The plan with every measurement is docs/plans/PLAN-2026-09-16-agent-quality.md.