Where Capability Stands

The tier depends on the cost of failure, not on how good the demo looked. Capability progress and reliability progress move at different speeds.

Whether a capability is usable depends on what happens when it is wrong, not on how impressive the demo looked. That sounds obvious, yet when a team debates whether AI can do a piece of work, the second standard is almost always the one in the room.

After reading, you should be able to answer:

• How do you judge that an impressive demo is not yet dependable?
• Why can the same model solve competition math and miscount the rows in a table?
• What sits between "it worked once" and "it works reliably ten thousand times"?

The tier comes from the cost of being wrong

Asking "is AI good enough" rarely produces an answer. A better move is to place the work in front of you: what happens when it is wrong decides which tier it can sit in.

• Tier one — run it as a copilot. Rewriting, summarizing, translation, extraction, code completion, drafting. What they share is that the output can be checked quickly and redoing it costs almost nothing.
• Tier two — run it behind guardrails. Knowledge Q&A, document analysis, bounded tool use, short flows. Supply authoritative material, limit which tools are available, and add evaluation, provenance and human fallback.
• Tier three — demo only for now. Long unattended runs, high-risk approvals, free operation on the open web. Permissions and scope both have to be narrowed hard, and a person must confirm the consequential steps.

How that first cut is made deserves a second look. "Use an LLM to process invoices" carries no tier on its own — reading the fields off an invoice into a form lands it in tier one; letting it generate and send payment instructions with nobody watching moves it to tier three. Nothing about the technology or the model changed; what moved the tier was the cost of getting it wrong.

One more thing is easy to miss: tier three says "not yet trustworthy", not "never usable." Capabilities in this tier usually demo beautifully, because a demo needs only one path that works. What they are short of is not ability, but what to do when the path fails.

Every cell has to be verified on its own

We are used to judging human ability along one continuous line: anyone who solves olympiad problems will handle percentages without trouble. The expectation holds for people because human skills are stacked one course on top of another. On a model it does not hold. Its strengths are shaped by the statistical distribution of the corpus — where text is plentiful and feedback dense, that region comes out sharp, and where humans would rank difficulty plays almost no part in how the ability is spread. So the combinations look like this: competition math comes out solved while the row count of a table comes out wrong; an entire module gets written while small text in a screenshot goes unread. The industry has a name for that shape: jagged capability. What the corpus favors and what it misses is the same fact stated from the other end. stanford-ai-index

None of this is a contradiction, and none of it means the model is being lazy. It yields one operating rule: capability is stored cell by cell, each cell has to be verified on its own, and the verdict next door does not carry over. "It handled the hard thing, so the easy thing must be fine" is precisely the inference that fails most often here. It is also why those three tiers cannot be assigned by demo quality.

There is a second gap. Producing something once and producing it reliably are never the same thing. A demo earns its keep once a single path works; a product has to catch every branch for the times that path does not. In between lie how the task gets split, how quality gets measured, how permissions are granted, how failures are undone, and who reviews — that stretch is the whole content of turning a model into a product. It yields no impressive demo, which is why it gets skipped most.

Reliability runs on a second, slower track

Every capability jump follows the same order: impressive demos arrive first, and evaluation, process and safety engineering arrive much later.

The reason is mundane. A demo costs days; reliability engineering costs quarters. One person can build the first; the second needs someone to define acceptance, collect failure samples, design fallback paths, and watch what happens after release. So in the period right after a new capability appears, almost everything you see is a demo — and the number of demos gets misread as evidence of reliability.

That is why every jump replays the same scene: a group concludes that this time is different and ships to production, then spends the first weeks after launch discovering that what actually blocks them is not the model being insufficiently clever, but that nobody defined what counts as done, how a failure would be noticed, or how it would be undone.

This is not an argument for pessimism about new capabilities. It only puts the order of judgment right: look at the cost of failure first, the quality of the demo second. The first decides how much you can let go; the second only tells you where the ceiling is.

The longer the chain, the more the success rate multiplies

Of the three tiers, agents are the most overrated by a wide margin.

No model suddenly acquired a will of its own here. What actually happens is that it repeats a set of actions inside one loop: pick a tool, see what came back, decide what to do next. The overall success rate therefore multiplies — hold a 95 percent chance on each step across twelve steps, and barely half of that survives to the end. A demo hands you one smooth path; the reality your product has to catch is that every step can go wrong and the damage compounds as it travels.

The way out is not waiting for a stronger model but cutting the chain shorter: break a long task into segments, put an automatically decidable checkpoint at each segment's end, and hand it back to a person when one fails. There is a precondition that gets skipped — verification has to exist before delegation can. Shorten the chain, automate the checkpoints, make failures undoable: with all three in place, an agent has moved up from tier three to tier two.

Two questions draw the boundary

When you cannot tell how far up the tiers a feature should go, run it past two questions. They come from the same root and are a simplified version of the three-second check:

Can the answer be verified? It holds up if a database can be compared against, a rule can be applied, or a person can check it. Conclusions that resist verification and carry serious consequences should not go straight to users.

Is failure reversible? A bad meeting summary gets regenerated; batch-editing customer invoices, clearing a production database or notifying every user need a gate in front. Let go where it is reversible; keep the human checkpoint where it is not.

With both boundaries drawn, hand over autonomy in sequence: proposals first, drafts next, execution only at the end. In engineering terms that order becomes two things. First, write "what counts as done" as a criterion a machine can decide — Defining Done First covers how to set it. Then split the handover into gates you can roll back — Launch Is Not a Switch covers how to release it. Autonomy is not one authorization; it is handed over segment by segment.

One last reminder: the line between "already usable" and "still a demo" keeps moving. The three tiers above are this article's own synthesis, not the conclusion of any report, and capability and reliability advance at different speeds. Before deciding, go back to your own acceptance criteria. stanford-ai-index-2026