What to Measure Before You Ship

A passing demo is not a launch case. Critical tasks need success rate, first-pass rate, all-pass@k, a quality floor, and cost per success — read at P85, not just the median.

A passing demo is not a launch case. Critical tasks need task success rate, first-pass completion rate, all-pass@k, a quality floor, and cost per successful task — read at P85, not only the median. One success in five and five in five look identical in a demo. In production they are two systems.

A demo is not evidence

The Anthropic incident is the cleanest counterexample. Three independent changes — default reasoning effort dropping from high to medium, a cache eviction bug, a verbosity prompt — each touched a different slice of traffic on a different schedule, and together they looked like a broad, inconsistent regression. The investigation began in early March, and the signal was hard to separate from ordinary variance in user feedback; neither internal use nor evals reproduced it anthropic-postmortem.

The lesson: the sum of component correctness is not system correctness. That is the only reason launch metrics exist — you need to ask the same question repeatedly against real traffic.

From the other direction: engineers on Hacker News dug into the claim of building a browser from scratch in a week and found code copied almost verbatim from Servo and stylo, quirks_mode() character for character, while the ACID3 screenshot required JavaScript enabled — that VM may never have run. The community landed on Roark66's summary: the path that works is a human-machine hybrid hn-cursor-browser.

Six metrics, read together

Separate three kinds of done: finished first try, finished after retries, finished after a human took over. Their costs differ by an order of magnitude, and merging them books human labor as success. Report at least the first four for critical tasks, watch quality and cost per success together, and never treat lowest token count as the target. One naming note: all-pass@k here is Pass^k in Define Done Before You Write Code, the same measurement under two names — one failure in k disqualifies the task — so do not track them as two separate metrics.

Distribution, floor, and the honesty gap

Report average performance, quality floor, and variance together. One successful run or a single judge score is not enough.

The long tail is where friction hides. One team drove median delivery time from 19 days to 9 while P85 sat at 52. Read the median and you believe the process got twice as fast; read P85 and you learn most of the friction never moved.

Then the honesty gap: self-reported results have to be compared against independent verification, and the comparison recorded. An agent claiming completion is not evidence the environment changed. Keep outcome — the final state of the world, whether the row exists — separate from transcript, the execution log. Only outcome counts as done.

Classify smoke failures four ways

Unclassified smoke failures send effort to the wrong place. CAPABILITY_FAILURE: fix the agent, tool, knowledge, or rules. INFRA_FAILURE: fix the environment and re-run, leave the prompt alone. TEST_DEFECT: fix the fixture or grader, not the agent. NON_DETERMINISTIC: add trials and let the distribution speak. Pin test data, dependencies, and versions, or you cannot tell the second from the fourth.

Confidence from evidence

Confidence has to be assembled from verifiable signals: evidence coverage, schema validity, deterministic checks passed, tool execution success, calibrated score. A model's self-reported confidence must not decide launch or trigger a side effect, and fluency must not stand in for correctness — the easiest signal to fake.

Fix the order: schema, types, static analysis, deterministic business validation; then execution, tests, state verification; then an independent grader; then human review. Anything a deterministic check can cover should not go to a grader. Define thresholds for automatic delivery, review, and stop per risk level.

A model swap moves more than the score

The most common launch-time decision is replacing a model: a new one ships, ranks higher on public benchmarks, and costs less. The question is not whether it is stronger. It is how much better it is on your specific tasks, and what switching costs you ai-agent-book.

Only an eval harness answers that. With one you run your own dataset in hours and compare success rate, tool-call correctness, latency, and cost. Without one you wait for production feedback, which tells you something broke and never what you could have had. The answer is often not a wholesale switch but a split: move the simple traffic to the cheaper new model, keep complex multi-turn orchestration on the old one where the new model's success rate falls. The precondition is the same as everywhere else — the gap has to clear the noise band first, see Verification Must Keep Up With Generation, or you are redesigning your architecture around random fluctuation.

One layer sits outside every score: what a model does by default. Given the same code task, one model explores the repository widely before touching anything; another locates a spot from thin local evidence, edits, and reconciles with test feedback afterwards. The first prices early edits as the bigger risk, the second prices "read one more file" as the bigger cost. A swap that leaves pass rates flat has not left behavior flat — exploration width, diff size, and retry counts move with it, and those three are where cost and stability come from ai-agent-book.

What the industry numbers say

McKinsey's State of AI survey from November 2025 found 62 percent of respondents experimenting with agents, but no more than 10 percent scaling them in any function. Gartner predicted that by the end of 2027 more than 40 percent of agentic AI projects will be canceled, citing escalating cost, unclear business value, and inadequate risk controls langchain-what-is-agent.

Between experiment and scale sits a whole infrastructure layer: observability, evals and datasets, sandboxes, access control. The common mistake is underestimating it and treating "it runs" as "it ships".

The trade-off is real: repeated trials, retained traces, pinned environments — not worth it for a throwaway script. Take one critical task, run it ten times, and write down success rate, first-pass rate, all-pass@3, P10, and cost per success. If you cannot produce one of them, your observability is unfinished.