Define Done Before You Write Code

Eval-first is not writing tests first. It is deciding which product decision the evaluation supports, then choosing metrics and graders.

Eval-first does not mean writing tests before you write code. It means answering one question first: what product decision will this evaluation support? Until that is settled you cannot choose metrics or graders, and every iteration optimizes a target nobody defined.

Name the decision first

When GitHub used an LLM to cut false positives in secret scanning, the real question was not whether the model could classify a string correctly. It was whether the system could cut alert noise while holding the recall floor. The first is answered with one accuracy number; the second forces you to declare where the safety line sits before you start. The result was a 95% reduction in false positives with recall held at the guardrail. Note the relationship between the two numbers: recall is not a peer metric sitting next to precision, it is a constraint that vetoes the change when crossed. github-evaluate-llms

The usual failure runs in the opposite order. Collect samples, write prompts, produce a score, then ask whether the score is high enough. With no decision attached, that question has no answer. Anthropic describes exactly this cliff. Manual testing, dogfooding, and intuition carry a team a long way, and then it collapses at production scale: users report the agent got worse, the team can only guess, real regressions cannot be separated from noise, and debugging becomes reactive — wait for a complaint, reproduce by hand, patch, and hope nothing else broke. anthropic-evals

Three tiers that do not trade against each other

• Primary output: the one or two metrics that drive the decision, such as the size of the precision gain. Improvement has to show up here.
• Safety constraints: allowed to degrade only inside a predefined threshold, and crossing the line is a veto. Recall lives here.
• Operational guardrails: latency, cost, reliability. These decide whether you can ship at all.

Not interchangeable means this: when an experiment doubles the primary output but breaks a safety constraint, the verdict is do not ship, not let us weigh it. A guardrail that accepts trade-offs is not a guardrail. github-evaluate-llms

Capability evals and regression evals

A capability eval asks what the agent can do well. It starts at a low pass rate, and that is fine — it gives the team a hill to climb. A regression eval asks whether the agent can still do what it used to do. It should sit near 100%, and any drop is a regression signal. The two connect over time: a capability eval that reaches a high pass rate graduates into the regression suite, which guards that capability while the team moves to the next hill.

Confusing them costs you in both directions: early capability scores look like a project out of control, and 95% on a regression suite looks acceptable. anthropic-evals

One pass rate, two meanings

A pass rate still does not say which claim it makes: "at least one run in k succeeds" or "no run may fail". The first is Pass@k — run the same task k times, it passes if any run passes; with continuous scores you take the best run, called Best@k. The second is Pass^k, read as Pass consecutive k: k consecutive runs, every one must pass, and none may trip a veto item such as safety, compliance, or hallucination ai-agent-book.

Take a single-run success rate of p = 0.6 with k = 5. Pass@5 = 1 − 0.4⁵ ≈ 99.0%, which reads as almost always succeeding; Pass^5 = 0.6⁵ ≈ 7.8%, which says five clean runs in a row is still hard. Same model, same task, a gap of more than an order of magnitude. The first number measures the ceiling you can reach while exploring, and it is what makes a demo look like a breakthrough. The second is what payments, refunds, permission changes, and production deploys actually ask for.

So state which k you used: k independent samples of one task, or k consecutive tasks on a production line. For operations with side effects, "retry until success" is not a valid sampling method; sample in a sandbox or a rollback-capable environment and count every failure into the reliability number. Pass@1 covers ordinary traffic, Pass^k is for critical operations, and Pass@k or Best@k is for exploratory work. Whatever sits in the pass-rate cell of your report, say which of the three it is.

Match the grader to the job

The division of labor: hard metrics go to deterministic graders. Is the row in the database, is the JSON valid, did the agent call a forbidden tool. An LLM judge rates semantic quality and nothing else — never exact counts, arithmetic, or schema conformance. Sample high-confidence judge outputs on a schedule to catch systematic errors, and version the judge prompt itself. github-evaluate-llms

SWE-bench Verified shows what a deterministic grader looks like when it is written well. It splits "the bug is fixed" into two separate propositions. FAIL_TO_PASS tests must fail before the patch and pass after it, which proves the problem is gone. PASS_TO_PASS tests must pass on both sides, which proves nothing else broke. Check only the first and an agent can delete or loosen the assertions in its way; check only the second and you have not checked anything. With both, "fixed" and "not broken" become two independently provable conclusions. The suite also confirms the tests themselves are stable, dropping flaky ones rather than letting them decide the verdict ai-agent-book.

Change one variable, and watch your wording

Test prompt revisions and model upgrades separately before combining them, and record prompt, model, dataset version, and system config so a run is reproducible. When you use production data as an eval set, audit where the labels came from first. Production labels usually record workflow outcomes rather than ground truth: an alert marked dismissed might mean a rotated credential, an accepted risk, or a misclassification, all collapsed into one label. Take that at face value and your conclusions will be nonsense.

Wording belongs on the engineering list too. In one team's work on evolving SKILL.md, swapping the single word vulnerability for risk dropped accuracy from 89.3% to 62.1% — 27 points, because risk has a far wider boundary than vulnerability. Use narrow words for core judgments, and keep a trap-word list per task.

Andrew Ng names the disciplined evaluation and error-analysis loop as the most important trait of an AI engineer. Not a technique: the habit of concentrating effort where it is most likely to pay off, so progress is systematic rather than random. It also means evaluating your evals, since the right mix of deterministic checks, judges, and humans shifts by project and even by project phase. andrew-ng-ai-skills