Evaluating Agents

No eval means running blind. A minimal acceptance plan you can actually run.

Plain question-answering has a right answer, so it can be graded automatically. An agent is multi-step, has no single correct answer, and its process can drift — which is why most teams skip eval altogether and go by feel. Between "it feels better" and "it is better" sits a runnable eval. anthropic-agents

Symptoms you'll recognize:

• You tune a parameter and it seems better, but you can't say why.
• You ship a new feature and can't tell whether it hurt other tasks.
• It runs several more rounds than before, and you don't notice the bill.

Why eval is hard

Three problems:

• No ground truth. "Summarize the action items from this meeting" — what is the correct output? There is no single answer, so machines can't grade it.
• The process matters. The result was right, but it took 30 rounds, 8 tool calls, and nearly deleted the wrong data. Is that a success? Looking only at outcomes, you'd miss it.
• Changes have a wide blast radius. An agent is a system — editing one prompt or adding one tool can shift the behavior of the whole flow. Without eval you can't know whether a change made other tasks worse.

These difficulties are matters of degree, not reasons to skip it. The smallest eval is a hundred times better than no eval.

Three layers of metrics

Outcome layer: how well did the task get done. This layer sits closest to the product goal. Since there's no ground truth, score it by hand — or let a stronger model score and spot-check. Scores can have several dimensions: completeness, correctness, whether it caused side effects.

Process layer: how did it do it. This layer usually exposes problems earlier than outcomes do. Read the run logs: which tools were called, in what order, how many retries, any detours, any actions that were blocked. Process metrics can be collected automatically and are far cheaper than outcomes.

Cost layer: what did it cost. Rounds, total tokens, duration, tool calls. Two implementations of the same feature can differ by 10x in cost. Read cost together with outcome and process — a fast, cheap, mediocre solution and a slow, expensive, perfect one are two different product decisions.

The rhythm: the cost layer runs often and automatically, the process layer runs on every change, and the outcome layer gets judged periodically by humans.

A minimal eval set

Don't wait for perfection. Start here. openai-agents

• Prepare 20 representative tasks covering three kinds: common tasks (the 80% users actually do), edge cases (missing parameters, abnormal input), and dangerous scenarios (ones that trigger high-risk actions). Better a few that represent reality than many picked at random.
• Write a scorecard per task. Not a model answer, but "what counts as acceptable": did the result land, was the process acceptable (say, one retry allowed, no endless retrying), and what's the cost ceiling (say, at most 10 rounds).
• Run once and record all three layers. Then the critical step: keep that record as a baseline. Without a baseline, no future change can be compared.
• Rerun after every change. Prompt edits, model swaps, new tools, loop logic changes — rerun the 20 tasks and compare against the baseline, watching all three layers.

When to rerun

Not every change needs a full eval. Grade by blast radius:

• Small change (one prompt line, one parameter): run the process and cost layers. Cheap, automatic, frequent. Spot-check a few key tasks on outcomes.
• Medium change (new tool, loop logic, model swap): full three layers. This kind of change is the most likely to "fix A and break B," so all 20 tasks.
• Large change (new architecture, multi-agent reorg): full eval plus post-launch monitoring. Keep watching process metrics in real logs after launch — many kinds of drift only appear in production traffic.

One last rule: eval is part of the product, not a gate before launch. Make it routine — a baseline, comparisons, and records. Drift isn't the disaster; drifting without knowing is.