Four Layers of Evaluation

Two agents can both get the task right while one is reproducible and the other got lucky. Scoring only the outcome files them under the same score.

Two agents both finish the task correctly. One plans cleanly, takes four steps, and holds up on the next batch. The other thrashes and stumbles into the answer. Score only the final output and they tie, but the first success is reproducible and the second is luck. An eval that records only the outcome files two different levels of engineering under the same number.

Getting it right is not the same as being right

Start by separating two words. Outcome is the final state of the environment: does the booking row actually exist in the database. Transcript is the full record: outputs, tool calls, reasoning, and intermediate results. An agent saying done is not the environment having changed, and acceptance goes by outcome. This deserves its own check, because completion logic routinely trusts the model's claim.

Long-horizon agents change the unit of evaluation too. Short tasks fit the (query, ground_truth, answer) triple; long-horizon tasks need (prompt, expected_behavior, trace). expected_behavior describes the behavior the agent should reach, not just the final answer, and the trace supplies the path to evaluate. You are judging whether the thing got done and how, not how well it was phrased.

Frontier models press against the edge of that definition. Opus 4.5 found a policy loophole on the tau-squared benchmark and booked the flight by breaking the rules as written — a failure by the eval's own definition, and a better outcome for the real user. That is not a bug in the eval. It is a reminder that static evals miss creative solutions and then score them as failures, which is why human sampling cannot be replaced by automated scoring. anthropic-evals

Another tau-squared trajectory photographs the opposite error. The telecom agent policy opens with "you should only make one tool call at a time", yet message 4 issues two, looking the customer up by phone and by name together. Both env_assertions pass, reward = 1.0, and the verifier stays quiet, because that task's reward_basis aggregates the final state only. This is not an oversight in the framework. It is the price of a binary reward: process granularity traded for a single number comparable across models. The two cases mirror each other — one scores a creative solution as a failure, the other scores a rule-breaking step as a perfect run. Production evaluation usually has to do more than return a verdict; it has to say which step went wrong. ai-agent-book

The four layers

What you evaluate is a compound system of model, harness, tools, and process, not a single model. Meituan's Turing team uses a four-layer structure worth copying: meituan-agent-eval

Attribution has to follow the same layers: a report that hands over one score has to say whether the failure came from planning, a tool, the environment, or a skill.

The four layers also work in reverse, as the schema of a test case. In tau-squared, each task's evaluation_criteria holds exactly those checkable dimensions: env_assertions for the end state (mobile data on, speed test above 200 Mbps rated excellent), actions for whether the key calls really happened, and communicate_info plus nl_assertions for whether the user was told what they needed to be told. reward_basis then decides how those dimensions aggregate into one score ai-agent-book. The difference is concrete: the layers are not four columns you add to a report afterwards, they are four fields you fill in while writing the case, and the score is only their aggregate view.

The process layer also needs an environment that withholds something. Those same tasks keep known_info to three items — name, phone number, country — while the real fault (airplane mode on, data roaming off) sits outside it, so the agent can only reach it by asking and guiding. The simulated user has its own tools (check_status_bar, toggle_airplane_mode, run_speed_test), and the fault is on the side the carrier database cannot see. Grounding is the other half of the bargain: every answer about device state has to come from tool output, otherwise the simulated user nods along when the agent asks whether it is fixed now, and the evaluation degenerates into two models confirming each other ai-agent-book.

Bridge metrics: connecting business and agent numbers

Offline scores climbing while business impact stays unexplained is the fastest way to lose trust in an eval program. The usual cause is a missing middle layer. Business metrics such as DAU and retention do not map directly onto agent-layer metrics such as intent accuracy and retrieval effectiveness; system metrics like recall and click-through have to connect them. Someone who understands the business process must help build it, or you cannot answer why a business metric moved or why a stronger model produced no business gain. meituan-agent-eval

Human agreement is the precondition for machine scoring

Do not hand subjective criteria to a model. Break a vague metric into several rubrics, then force each toward a binary: yes, no, or unknown. The share of unknown answers probes the rubric itself — a high share means the question is ambiguous and the definition needs fixing.

Then measure two things: how often humans agree with each other, and how often the machine agrees with humans. Machine scoring does not scale until both clear a credible threshold, around 85% and 90%. Below that, machine output is machine labeling, not automated evaluation. Meituan's Beam team took human-model agreement from 62% to 92% with exactly this binarization work. meituan-agent-eval

Write scoring questions as formal specs

A rubric is not "describe what good looks like"; it is a spec handed to the judge. Google's four design principles google-rubric: atomic, non-overlapping questions — a compound question ("does the response contain metadata and is the output JSON") forces the judge to guess which clause matters; split it into independent TRUE/FALSE items, and never test the same concept twice, because overlap double-penalizes a single error. Objective facts only — write the spec in RFC 2119 terms (MUST / MUST NOT), test negative constraints (check "did not recommend a deprecated feature", not "used best practices"), and stay away from intent or quality, concepts that demand interpretation. Score only what the prompt explicitly asked for — if the prompt did not request citations, do not deduct for missing ones; grade the destination, not the path, and if you need to judge the process, have the agent emit its execution plan and grade the plan. Calibrate against a golden set before launch — have domain experts label first, run the judge against the human scores, treat misalignment as evidence the rubric is ambiguous, and iterate until they agree. The side benefit is concrete: scoring strict boolean facts is a simpler task, so a smaller and faster model can do the grading.

The judge prompt itself carries four engineering rules aliexpress-eval: one prompt scores one metric — mixed scoring measurably dropped both accuracy numbers; output reasoning before the verdict so failures are explainable; include pass/fail contrast examples as negative guidance; and enforce strict JSON output with a format template. When deciding pass or fail, one primary metric per dataset decides (end-to-end goes by completion rate, fuzzy-intent sets by clarification rate — asking beats guessing), with other metrics used only to locate weak spots, never all-metric AND — when one case fails several metrics, AND misreads a module weakness as an overall failure. When upstream failed, downstream metrics are marked skip, not fail: if the routing decision was wrong, the retrieval metric no longer reflects the retrieval module. Hallucination eval draws the same narrow boundary: judge only whether the agent faithfully relays the knowledge fragments, tool returns, and conversation history it obtained — not whether the source information itself was correct. The agent is an information processor, not an information authenticator.

One owner decides; disagreement becomes a strategy branch

When the standard is not aligned, you cannot tell whether a metric moved because the system improved or because the standard shifted. So one person owns the standard and settles disputes. One decider beats ten voters.

Disagreement between experts should not be forced into consensus, though. Persistent disagreement usually means the business supports several good strategies — aggressive and patient outreach can both close sales. Build the shared part into the eval system, turn the contested part into separate style or strategy branches, and evaluate each on its own test set rather than averaging the disagreement away as noise.

Do not design an elaborate system on day one. The more complex a metric, the harder it is to execute and align, so a turning data flywheel matters more than a complete taxonomy. Meituan's fulfillment agent grew from just over 20 metrics to nearly 200 in a year, fed by bad cases that expose capability limits and good cases that define quality. It grew; it was not designed. meituan-agent-eval

Closing the loop: from failure data to system evolution

An eval that stops at reports and scoreboards is a checkpoint after the fact. It cannot make the agent stronger.

What an enterprise should expect from evaluation is that every failure it surfaces becomes a protective asset in production.

On one side, for violations the risk layer catches — an agent trying to run an unauthorized delete, say — the system should not simply dock a point in a report. It should extract the pattern of that action and turn it directly into a deterministic Hooks rule at the pre-tool gate, so the same risk cannot occur again in production.

On the other, for inference errors and tool misuse the process layer catches, the framework should cut the context slice around the first deviation, add it to the regression suite, and hand it to the offline experience-learning pipeline for a targeted strategy note.

The mechanism that feeds evaluation back into governance and rules is worked out in full in Experience Saved Is Not Experience Learned. Only when the loop runs end to end — the run exposes a problem, evaluation pinpoints it, a rule is distilled, regression admits it — has the four-layer structure delivered its engineering value.