Tier Your Recovery, Then Circuit-Break
Recovery escalates by how transparent it is to the user, and every path needs a circuit breaker grounded in production data — otherwise recovery becomes the next failure.
Recovery is not "try again". It is a mechanism with tiers, ceilings, and exits: solve it at the lowest tier if you can, give every path a circuit breaker, and let the error-handling logic never manufacture new errors. Where do the thresholds come from? Production data — never guesses.
Three tiers, ascending transparency
Tier one, silent retry: the default for retryable errors, decided by two details — exponential backoff plus random jitter so a synchronized herd of clients does not cause the second congestion, always honoring the server's wait-time hints; and the foreground/background split, where main-loop failures are retried while auxiliary background calls like title generation fail fast, or background retries eat the main path's quota and create retry amplification ai-agent-book-5. Tier two, degrade and resume: when truncated at the output cap, first resend with a silently raised cap, then append a meta-instruction that continues from the cut; when the primary model stays overloaded, downgrade to a backup model — after stripping the old model's private format blocks, or the new model cannot parse history. Tier three finally exposes the error to the user, with the recovery actions already attempted. Tool-layer errors take a different road: do not end the session — feed the structured error back into context and let the model correct itself next turn; the more concrete the error, the higher the self-correction rate.
Breaker ceilings come from production data
The previous article demanded a counter per recovery path; this one spends them. Every path needs an explicit breaker ceiling — abandon compaction after several consecutive failures, fall back to asking a human when the permission classifier keeps failing, cap continuation attempts at a fixed number of rounds ai-agent-book-5. How was Claude Code's compaction breaker set to "3 consecutive"? Real session statistics: one session failed on that path more than three thousand times in a row; this class of wasted retry alone burned roughly 250,000 API calls per day worldwide; over a thousand sessions hit 50+ consecutive failures. Three is the empirical knee between "most faults have already recovered" and "more retrying is hopeless" ai-agent-book-5. Your own knee can only be read off the distribution your counters collected.
The recovery logic can cause the fault
More hidden than any single-point failure is the death spiral: logic inside the error path calls an LLM again, fails again, and the cascade starts. A real incident: the agent stopped because the context overflowed, which triggered the "auto-commit on exit" stop hook, whose LLM-generated commit message overflowed the context again, which triggered the hook again ai-agent-book-5. Two defenses: forbid any side-effect logic that calls the model again on an error path — losing one auxiliary feature like automatic memory extraction is acceptable — and use a recursion-depth counter to detect and break residual chains. Above all single mechanisms sit the global termination and escalation conditions: maximum iterations, session budget, human escalation past a consecutive-failure threshold. A retry ceiling is an architecture decision, not folklore — what the system does after the fourth failure decides whether it fails safely or becomes a hidden source of wrong output: user-facing systems usually degrade gracefully, since a safe but ordinary reply beats an error; internal pipelines should fail explicitly, since downstream needs to know the retries are exhausted; for high-risk tasks, auto-returning anything after three consecutive generation failures is almost always the wrong call agents-in-action-7.
Intermediate errors stop at the loop
The core principle in one line: the boundary of error handling is the whole recovery loop, not a single request ai-agent-book-5. Until recovery is declared impossible, intermediate errors do not reach consumers — neither users nor event subscribers: hold them during recovery, stay silent if it succeeds, present them together only when every method has failed. Skip this and users see error toasts for a fault that already healed, and downstream systems mark transient blips as task failures. The cost view agrees: held retries still burn tokens — Attribute Cost Down to Every Step warns that a total figure cannot tell "the agent improved" from "the agent started retrying more", and the recovery loop is exactly the spending that hides there.
Switching vendors is not changing an endpoint
When the primary model stays down, another vendor must carry the trajectory to the end. The obstacle is not endpoint URLs: thinking content pairs readable text with vendor-signed credentials, and text survives the move while credentials die at the border — you can take the words, not the receipts ai-agent-book-5. Takeover plans must be designed against the strictest vendor, with an escape hatch: rewrite past tool calls into plain narration — the model no longer treats them as real calls, but can keep moving. The design principle: store trajectories in a neutral format — thinking split into portable text and non-portable credentials, tool calls as name and arguments only, identifiers regenerated per target vendor, credentials always dropped on switch. The same neutral trajectory also serves evaluation replay and experience extraction; failover is just its cheapest use.