Model Selection

Most requests never need the priciest tier, but the primary model will fail. Tiering, fallback, format normalization, and logging all belong in one thin wrapper.

Not every request deserves the most expensive model, and a backup only counts once it has actually run. Both problems have to be solved in the same layer — otherwise one outage takes back every dollar your tiering saved.

After reading, you should be able to answer:

• Which few things should you compare between providers, instead of who scores higher?
• How do you tier by task, and why is dynamic routing usually unnecessary?
• What must a backup path be verified for, and where does a failover break first?
• After changing models, how do you know quality did not drop?

Do not compare providers on scores alone

Leaderboards turn over every month, so memorizing them buys nothing; and model capability is the visible part, which anyone can test. What decides whether you get burned is the rest of the picture.

One, how data is handled. Whether inputs are used for training, how long they are retained, and whether that can be put in a contract. This decides whether you can take on customer data at all.

Two, regional availability. Where your users are and where compliance requires the service to be. Data crossing regions is simply a non-starter in some industries.

Three, whether the model name is stable. The name you pinned today — will it still point at the same thing in six months?

Four, whether a version can disappear. Whether you can pin a specific version, when it retires, and how much notice you get.

The last two get ignored most because nothing goes wrong in normal times. But your product rests on the assumption that the model will keep behaving the same way, and only the provider can guarantee that.

Capability gets compared too, but not by reading a board — by running your own cases through it. Which model is cheaper, which is stronger on your scenario, comes out of that one measurement and nothing else.

How you reach a model decides who is responsible when it breaks

The same model reached four different ways differs not in features but in responsibility.

• Straight from the provider. Version, quota, and incident handling all sit with one party, at the cost of a separate account and settlement per vendor.
• The same model through a cloud platform. One more layer of procurement and contracting, in exchange for invoices and compliance paperwork — and one more link in the chain.
• Weights from a model hub, run yourself. Responsibility is entirely yours, and what you get back is control.
• Through an aggregator. No vendor-by-vendor integration, but your requests pass through a third party you do not fully control, and the route can be cut upstream at any time.

There is really one question to settle: is the version you pinned the one you actually get, and whose hands does your data pass through. The more links in the chain, the slower you locate a failure.

Not every request deserves the most expensive model

Many teams pick one strong model and run the whole product on it. It is the easiest setup, and the price of that ease is that it is both costly and brittle: costly because a large share of requests never reach that tier, brittle because when that single model falters, the whole product stops with it.

Start from one premise: the requests arriving at your product were never one kind of thing. "Pull the amount and tax ID off this invoice into fields" and "read three years of sales history and argue which product line to cut next year" sit orders of magnitude apart in difficulty, and in practice they are often handed to the same model.

Tiering comes down to one rule: if a smaller model will do, do not reach for a bigger one. The vendors' own engineering guides hold the same line: what a simple approach can solve should not be made more complex. anthropic-agents

• Light tier: classification, tagging, field extraction, format conversion land here. Such tasks have few possible answers and an unambiguous right and wrong, so a small model is enough — and it responds faster at a lower price.
• Mid tier: Q&A, summarization, content generation, everyday rewriting. This is the workhorse, and it covers most scenarios.
• Heavy tier: only multi-step reasoning, complex code, and trade-off analysis need it. It calls for a reasoning model, and its share of total volume should stay capped.

How do you decide which tier a request belongs to? First ask whether it clears the line, then ask who does it better. Let the light tier try it; once it clears the line, stop there — the small set of real cases you built is exactly what serves as that passing line here.

The payoff is usually large: move your highest-volume simple features to the light tier and the bill drops by a large block, with users noticing nothing.

Route by feature entry, not by guessed complexity

"Smart routing" is surplus to requirements in most settings. Static assignment by feature entry solves it: invoice field extraction always goes light, support-reply polishing always goes mid.

Dynamic routing earns its place in exactly one situation — when the requests entering a single entry point vary widely in complexity. And even then, the routing decision is itself often a model call, carrying its own expense and its own chance of getting it wrong; the savings have to cover that cost before anything else.

Static assignment brings a second benefit: with the input structure of each pipeline pinned down, prefix caching holds more reliably. Dynamic routing scatters one entry's requests across different models and the hit rate drops with it. Tiering and caching therefore pull against each other, and belong in one decision rather than two.

Where the thin wrapper earns its place

Everything above — tiering, switching, format normalization, logging — happens in the same place: one thin wrapper in your own code.

It exists not for architectural elegance but because business code should not be wired directly to one vendor's interface. Once it is, changing models is not a one-line edit; it is a sweep through the whole codebase.

The wrapper carries at least three responsibilities:

• One entry point. Business code says "do this task," not "use this vendor."
• Format normalization. Whatever any model returns, business code sees one structure. A backup model's format differences must be absorbed here, never leaked upward.
• Logging. Which model each call used, what it cost, and whether it failed over.

One observation along the way: most providers have converged on the same de facto request format, which makes "try another vendor" cheap. But easy to switch is not the same as safe to switch — similar shapes hide different behavior, and the same prompt can perform very differently across models. Only measurement finds that. What makes the wrapper valuable is that it turns both measuring and switching into low-cost actions.

A backup that has never run is not a backup

Model services rate-limit, time out, return errors, and retire versions. All of that is routine, not an accident.

Failover goes wrong more easily than you would expect. Three places it trips first:

The backup path has never actually run. It sits in the config, and nobody has checked how it behaves under your prompts. Come a real incident, you switch over and find its output shape differs from the primary's, and the parsing logic gives up on the spot. A backup nobody has run does not count as a backup, so the sounder arrangement is to let it carry a share of live traffic on an ongoing basis.

The two output formats do not line up. This is the most frequent one. Your parsing logic was written against the primary's output, and the swap changes it. Format constraints that hold on the primary are not guaranteed to hold on the backup.

The failover happened and nobody knew. The switch completes silently, leaves no record, and then someone reports that answers seem off today while the investigation goes nowhere. Every failover has to leave a trace, and the failover rate has to be visible in monitoring.

There is also a product decision to settle up front: do you tell the user about the failover? Where the fallback visibly harms quality, quietly carrying on is usually worse than being open about it.

Whether quality dropped can only be answered by a fixed case set

Adjusting a tier, changing a model, falling back — after each of these you come back to the same question: has quality got worse? That question cannot be settled by feel.

The method is plain: keep a fixed set of cases. Draw the material from real requests — start with about thirty — covering the common situations and keeping a few especially awkward edges. Before any change to the model, run the set, then lay the results beside the previous round and compare case by case. To do this more systematically, there is a whole set of vendor tooling and practice worth borrowing. openai-evals

Three practical reminders:

A model change usually drags a prompt change with it. Vendors respond to the same prompt in very different ways. When the first round disappoints, revise the prompt for the new model before judging — writing it off immediately usually does it an injustice.

An average hides the problem. A flat average can conceal "most cases got better, a few fell apart." The cases worth watching are the worst ones; those are what users complain about.

Let a slice of traffic go first. A few days at small volume exposes real problems more readily than a hundred runs in staging. Launching is not a switch is the same rhythm.

That case set evolves with the product, so it is itself an asset to maintain — the evaluation set needs feeding.

What this does not imply

It does not imply that selection is a one-time decision. Models turn over every six months and prices move with them, so selection is something to redo every few months.

Nor does it imply that more tiers are always better. Every extra tier is another pipeline to evaluate, monitor, and prepare a backup for. The number of tiers should follow the differences between your tasks, not your anxiety about the bill.

There is really only one test: the same batch of requests, with quality held constant, at a lower bill. Fail that and any tiering scheme just adds maintenance surface. And making it work at all depends on cost being attributable in the first place — attributing cost to each step covers how to get there.

That closes out the methods for running it stably and cheaply. One class of things remains that it cannot supply: your own business, documents, and internal concepts — none of that is in its corpus. How to fill that gap starts with how much one request can hold.