Classify Failures Before You Count Them

The first judgment after a failure is not "retry or not" but "is this retryable": four failure layers need four policies, and counts mean something only after classification.

The first judgment after catching a failure is not "should I retry" but "is this worth retrying". A blanket "retry three times on error" treats rate limiting and an illegal parameter as the same thing: the former clears after a few backoff rounds, the latter fails identically however many times you send it — and both end up in the same failure rate. Runtime classification is the precondition for every count and every circuit breaker.

Failures happen in four layers

Sort failures by where they occur, because each layer needs a different response: back off for rate limits, repair pairing for missing results, compact for overflow, break out of loops ai-agent-book-5. What all four share is only the counting. Merge the four layers into one failure-rate curve and you have built a machine for wrong conclusions: tool errors tripling could mean a new model version, or just an upstream service rate limiting — the table cannot tell them apart.

The first call: is it worth retrying

Retryable errors — rate limits, overload, network jitter — become fine on a second pass. Non-retryable ones — illegal parameters, insufficient permissions, a tool that does not exist — retries just burn money; the input or the strategy has to change ai-agent-book-5. Production harnesses keep an error-to-recovery-policy mapping table instead of one vague "retry on error" rule. How to maintain the table: for every error code and exception type, record which layer it belongs to, whether it is retryable, which recovery path handles it, and which counter tracks it. Without this table, no circuit-breaker threshold downstream is meaningful.

Before counting, look for patterns

A single error is easy; the chain-reaction kind is caught by pattern detection. Two signals. Repeated call fingerprints — compute a fingerprint over "tool name + arguments", and the same fingerprint recurring is a hard signal of a no-progress loop. Per-path consecutive-failure counters — every recovery path gets its own counter, which is the next article's breaker threshold ai-agent-book-5. Why structural signals rather than error messages? Distributed fault tolerance splits faults into crash faults and Byzantine faults; agent failures are Byzantine by nature — an agent rarely stops; it keeps producing plausible wrong conclusions, and errors never announce themselves as errors ai-agent-book-10. So classification cannot wait for an exception to appear; it reads fingerprints, counters, and pairing checks.

The worst failure throws nothing

For streaming connections the most dangerous mode is not disconnection — that errors immediately — but the silent hang: the connection is established and the data flow stops, a pipe that is open with no water coming. SDK timeouts often cover only the initial connect, not the transfer, so production agents need an independent idle watchdog: no new output past the threshold means killed and retried. Generalize it: every long-lived connection needs a liveness signal, not just a connection timeout ai-agent-book-5. Trajectory integrity works the same way: when a tool call lacks its paired result, the harness repairs the pairing before injecting into context instead of handing a broken structure to the model or the user. One detail worth copying: some production agents run product mode and training-data collection mode side by side — product mode patches missing messages with placeholders, training mode refuses the patch, because synthetic placeholders poison training data. Same corruption, two policies: the double standard is itself an engineering decision.

Runtime and eval are one pair

The four smoke-failure classes in What to Measure Before You Ship — CAPABILITY, INFRA, TEST_DEFECT, NON_DETERMINISTIC — classify on the evaluation side to decide where effort goes; this article classifies on the runtime side to decide what to do about an error. One spirit, two domains: an unclassified failure number cannot guide any action. The audit is plain: do you have an error-to-policy table? Does every recovery path carry its own counter? Beyond connection timeouts, do long-lived streams have watchdogs? If any answer is no, your retry policy is still superstition.