Attribute Failure to the First Deviation
Attribution targets the first error that pushed the task off course; later errors are chain reactions. Where you locate it decides whether to fix harness, workflow or data.
An end-to-end eval tells you success or failure and answers nothing else — not "why", not "from which step". For eval results to actually drive fixes, every failing trace needs attribution: the main error category, the step where unacceptable behavior first appeared, the tool call or model output involved, and evidence someone else can re-check. The object of attribution is the first error that pushed the task off course — later errors are usually chain reactions, and the last error message is not the root cause ai-agent-book-7.
Three signal sources, one principle
Production problem cases come from three signals: users correcting the agent explicitly, thumbs-down and other negative feedback, and issues found afterwards by state checks and verifiers. All three feed one taxonomy — the categories may run into the hundreds, but the rule for picking the primary cause is a single sentence: take the earliest error that explains the subsequent failures, and keep the rest as secondary ai-agent-book-7. The nine common error types (requirement misunderstanding, process violations, tool misuse, hacking the verification environment, incomplete edits, information feedback, non-functional regressions, execution exceptions, premature stopping) each have their own way of locating the first error, and all of them reduce to the same instruction: walk backwards from the first deviation rather than forwards from the last exception. How the record itself is structured and turned into a replay case is already covered in Turn Failures Into Engineering Assets; this article is only about the localization.
Right action, wrong report
One failure class is hidden by aggregate scores more reliably than any other: the environment state is completely correct, and the information handed to the user is wrong. In the published τ²-bench baselines, the runs carrying an information-notification requirement failed 240 times out of 704, and 162 of those failures were notification errors — 80 cases where the environment state was right and only the words were wrong, one third of all failures ai-agent-book-7. An AndroidWorld T3A run log contains the textbook sample: the task was to enter the expenses from expenses.jpg into a bookkeeping app, and the agent finished authorization, search, opening the image, entry by entry, and saving in 32 steps with no step ever returning an error, then declared completion — while the checker reported that no record existed, and the log showed one entry, unrelated to the four it claimed to have written. Reading step by step: the thinking at step 8 says "I cannot see the image content", and the agent neither stopped nor reported it; at step 11, four expense entries appear in its notes out of nowhere; every later step is built on those four hallucinations ai-agent-book-7. The first error is at step 8 — which raised no error and was not even a tool call. The root-cause judgment is the part worth copying: T3A is a text-only agent whose observation space holds an element tree and no image pixels at all, so the cause is not "the model cannot OCR" but "the harness has no observation channel for it". Record it as a model capability problem and the next step is swapping models or building OCR training data; what actually needs doing is adding a channel. Where you attribute decides where the repair budget goes.
Layer by layer: from bytes to tokens
"The model copied it wrong" does not qualify as a root cause either, unless every layer in front of it has been ruled out. The standard move for an old_string match failure: keep the raw byte hash, the Unicode code point sequence, and the tokenizer token ID sequence for the same string, then find the first difference along the chain file bytes → tool return → harness serialization → model context → token output → decoded string → JSON parsing → tool matching ai-agent-book-7. Minimal probes cover direct restatement, extraction from a long context, placement into a tool argument, and choosing between similar strings, with whitespace, newlines, backslashes, Unicode combining characters and low-frequency tokens as the traps; the metrics are byte-exact, code-point-exact, token-exact, and the position of the first divergence. The routing rule is one sentence: if the direct probe outputs correctly and the tool call fails, fix the tokenizer, the serialization, or the tool protocol; the case becomes training data only when the first difference appears in the model's own output — and misjudging it means feeding the model dirty work that the harness should have absorbed.
The first error is also a training label
The same attribution record keeps working on the parameter side: the first erroneous step and its error category are the direct source of process supervision negative labels (PRM), and the decision boundary just before "about to claim completion" can be cut into preference pairs — rejected is the premature finish, chosen is running the tests first and then concluding ai-agent-book-8. In the premature-stopping case, attribution places the first error at the step where a conclusion was drawn without evidence: reading the code and editing it before that point were both fine, and the two trajectories share a prefix but reach opposite endings ai-agent-book-8. This is also the engineering ground that makes the Outcome/Transcript split in Four Layers of Evaluation hold up: without first-error localization you only have a score; with it, a score decomposes into actionable fixes.