Prompt Caching

A cache only reads the prefix: it discounts only when the opening is byte-identical, and a reorder voids the whole thing. So prompt order is an economic decision.

Two teams running the same prompts on the same model can end up a factor of two apart in cost, and the difference is usually one thing: the cache hit rate. And whether you hit is not decided by your content. It is decided by your order — the prefix must be byte-identical to be discounted, and a reorder voids the whole thing.

After reading, you should be able to answer:

What earns the discount, and under exactly what conditions does a hit happen?
Which content belongs at the front and which must go last?
What do you give up by pinning the prefix?
How do you find out how much of the discount you are actually getting?

A hit reads the prefix, and demands an exact match

The mechanism is: the opening stretch of your request is compared against a previous request, the part that matches does not need recomputing, and so it bills at a discount. The discount can be large, which makes this the most direct cost lever available. anthropic-prompt-caching Both vendors' docs describe the same mechanism; the only difference is how you switch it on. openai-prompt-caching

Two words in that description need pulling apart.

"Prefix" means only the opening stretch qualifies. Matching runs contiguously from the start, so any mismatch anywhere along it means nothing after that point gets the discount.

"Exact match" means a single differing character breaks it. Not "the same meaning" — the same bytes.

Together those produce a counterintuitive conclusion: hits are not decided by whether the content is the same, but by whether the prefix is identical. Give the same batch of material, ordered A then B one time and B then A the next, and neither request hits — even though you changed nothing.

One more property: it is either a full match or none at all — partial credit does not exist. Everything from the first mismatched position onward settles at full price. So the question worth asking is not "how much of it hit" but "from where did it stop matching" — and that position is more useful than the hit rate itself, because it names the line to change.

Stable content first, changing content last

Organize every request in this order:

Fixed content first. System prompt, long-lived rules, fixed knowledge — identical every time, forming the stable prefix.
Changing content last. This round's question, temporary instructions, the material retrieved this time.
Nothing variable in the middle. Timestamps, random values, user identifiers, and strings assembled at runtime: let any of them fall inside the prefix and every request comes out different, so the cache never matches. Where the model genuinely does not need those fields, leave them out of the prompt.
Give variable content a fixed order too. Retrieved material differs every time, but its sort rule can be fixed — by descending relevance, for instance. Stable structure has value on its own.

One easily missed move: split frequently edited long-lived rules out of the system prompt into their own section after it. Then when you edit the rules, the section ahead of them still hits. Layered loading has direct financial meaning here — it is not only attention management, it is bill management.

How much you save comes from two numbers multiplied: how steep the discount is, and what share of your input falls inside the stable prefix. The provider sets the first; the second is yours to move.

A contract-review assistant shows this in practice. Every request carries the same review rules and the same output template, and that block is the bulk of the input, while each round adds only the clause under review. Leave the rules untouched and the large block keeps billing at the discount — how much of the input cost you shave off depends on that block's share, not on how steep the discount is.

In one line: order is a pricing question, not a layout one.

The cost is losing your freedom to edit the front

Pinning the prefix means giving up the freedom to change the opening at will. That is a real trade, not a free win.

The longer the prefix, the bigger the discount — and the more expensive it is to change. One edit restarts the whole prefix from scratch. So a feature whose prompt you rewrite daily keeps resetting its own cache benefit.

If the system prompt changes often, it becomes the part that always pays full price. In that case do not optimize the cache yet; settle the structure first. Once it stabilizes, the benefit stabilizes with it.

What you fix is the structure, not the content. The parts that must vary per task must vary. Hard-coding content that ought to change is self-deception — you save nothing and make the prompt worse.

So the test is simply: will this content change often? If yes, put it late. If no, put it early. That one question beats any trick.

One boundary matters just as much: the cache never touches the output tier. It discounts input only, and the output portion pays full price no matter what. A pipeline dominated by output can steady its prefix perfectly and still shave off only a corner of the bill.

The hit rate is one of the few metrics that explains the bill

Most providers show cached usage separately in the billing detail. Log it and compute a hit rate: cached input volume divided by total input volume.

A hit rate near zero for a long time means something in the prefix changes. The usual culprits are timestamps and dynamically concatenated strings.
A hit rate that swings usually means unstable ordering, or concurrent requests interrupting each other.
A high hit rate with no cost drop means most of your input was in the changing stretch all along, so there was little to discount.

Track this alongside cost, daily. It is one of the few metrics that both explains the bill and points straight at what to change.

What this does not imply

It does not imply that caching rescues every cost problem. Only the input tier can be discounted; if most of your bill is output, cut output first and come back to the cache afterwards — not the other way round.

Nor does it imply that switching to a cheaper model saves the same thing. Caches accumulate within one model at one provider, so changing model or provider means starting the accumulation over. That is why tiering models by task has to weigh more than the unit price: a slightly cheaper tier that cannot hold a cache can cost more in total.

The hit rate is also a classic "attributable before it is optimizable" metric. It has to be logged alongside the model, the pipeline, and the call count — otherwise you only know the bill went up, not which pipeline took it there. Attributing cost to each step is how you split it apart.