Selection and Fallbacks

Tier models by task, and auto-failover when the primary model goes down.

Many teams pick one strong model and run the whole site on it. It's the easiest setup, and also the most expensive and the most fragile — expensive because most requests never need that tier, fragile because when it goes down, everything goes down.

Signs you'll recognize:

• Most of the bill goes to trivial work like "is this message spam?"
• One model-service hiccup takes every AI feature offline.
• You switch to a cheaper model and discover after launch that some requests got much worse.

Tier by task

First, admit it: your requests are not all the same.

"Is this review negative?" and "write a module from this requirements doc" differ by orders of magnitude in difficulty, yet they often run on the same model. The first is fine on a cheap small model; the second needs a reasoning model. The principle is "use the smallest model that works" — Anthropic's agent-design guide makes the same point: start with the simplest option and add power only when it's proven insufficient. anthropic-agents

So assign a tier per task type:

• Light: classification, tagging, field extraction, format conversion, light rewriting. Small answer space, clear right-or-wrong — a small model is enough, and faster and cheaper too.
• Mid: Q&A, summarization, content generation, routine code completion. The workhorse tier for most features.
• Heavy: multi-step reasoning, complex code, trade-off analysis. Use a reasoning model, but cap its share.

How do you route a request? Most of the time you don't need smart routing — static assignment by feature entry is enough. "Spam detection" always goes light; "write the weekly report" always goes mid. Only when one entry genuinely spans wildly different complexity is dynamic routing worth it.

The payoff is usually large: moving the biggest-volume simple features to the light tier cuts the bill sharply, and users never notice.

Fallback is more than swapping

Model services fail. Rate limits, timeouts, errors, a version de-listed one day. This is normal, not an accident — a product must be prepared.

But fallback is easier to get wrong than it looks. Three traps.

Trap one: the backup has never really run. There's a backup in the config, but nobody verified how it behaves on your prompts. When you switch during a real incident, its output format doesn't match and your program crashes. The backup must be run for real on a schedule — ideally with some live traffic.

Trap two: incompatible output formats. The most common one. Your parser expects the primary model's format; the backup returns something different — especially with JSON output, where vendors differ a lot. So your wrapper layer must normalize format: whatever the model returns, business code sees the same structure.

Trap three: you don't know you degraded. Fallback happens silently with no record, then someone says "answers feel off today" and you search for hours. Log every fallback and watch the fallback rate in monitoring.

One product decision to make in advance: do you tell the user? If fallback clearly degrades quality, pretending everything is fine can be worse than being honest. Usually a line like "you're on the backup service; results may be less polished" beats letting users wonder.

How to verify nothing got worse

Every model change — a swap, a tier adjustment, a fallback — must answer one question: did quality drop? Feelings can't answer it.

The method is plain: keep a fixed test set. 20 to 50 real requests covering common cases and a few nasty edges. Every time you touch the model, run this set and compare side by side with the last run. It's the ruler for every model decision — the same point as the model-selection article, and the highest-return one-time investment in the whole loop. openai-evals

Three practical reminders.

One, a model change usually means re-prompting. Vendors respond very differently to the same prompt. When the first run disappoints, give it one chance with targeted prompting before writing it off — killing it immediately is often unfair.

Two, don't just watch averages. A flat average can hide "most cases improved, a few collapsed." Watch the worst cases — those are what users complain about.

Three, roll out gradually. Send a small slice of traffic to the new model and watch for a few days before going full. That finds real problems better than a hundred runs in staging.

The thread running through all of this: it only works if model calls are wrapped in your own layer. Tiered routing, primary/backup switching, format normalization, and logging all happen there. If business code is littered with each vendor's SDK, everything in this article gets expensive.