Cache Hits and Saving Money

A cache hits only when the prefix is unchanged. Reorder your prompt and the bill can halve.

Two teams can run the same model with the same prompts and end up with a 2x difference in cost. The difference usually comes down to one thing: the cache hit rate.

Symptoms you'll recognize:

• Your prompt embeds a timestamp or random content that changes on every request.
• User history is inserted in the middle of the system prompt, so the prefix differs every round.
• Your bill runs far above your peers' estimates, and you can't find out why.

How caching works

LLM providers commonly offer prefix caching: if the front part of your request exactly matches a previous request, that part is billed at a discounted rate (as low as a tenth of the input price), because the provider doesn't need to recompute the repeated content. anthropic-prompt-caching Some providers impose a minimum token threshold on what can be cached, but whatever the threshold, the principle stays the same: keep the prefix as long and as stable as possible.

Caching is automatic — you don't apply for it, but you do have to cooperate. Cooperation means exactly one thing: keep the prefix of every request as identical as possible.

Two keywords matter: prefix and exact match.

"Prefix" means only the opening part gets the cache. Any mismatch later in the request invalidates the whole prefix, because caching matches contiguously. "Exact match" means a single differing character breaks the hit.

What decides a hit

Three things break hits most often:

• Variable content in the prefix. Timestamps, random numbers, dynamically concatenated strings at the front mean every request differs and the cache never hits.
• A change inserted mid-stream. Putting user history between the system prompt and long-lived rules. The system prompt plus rules could have been stable; the inserted variable content voids the whole thing.
• Unstable ordering. The same content, ordered A then B one time and B then A the next — neither matches.

The cache is all-or-nothing, not partial: everything after the first mismatch is billed at full price. So ordering matters a lot.

Arranging the prefix

Organize every request in this order:

• Fixed content first. System prompt, long-lived rules, stable knowledge — identical every time, at the front, forming a stable cache prefix.
• Variable content last. This round's user question, temporary instructions, and retrieved material go at the very end.
• Avoid variable content in the middle. If user history must be included, place it after the fixed prefix and near the question — better a shorter but stable prefix than a long one that never matches.

More details:

• Drop pointless variation. Fields like "current time" or "user ID" change every time; unless the model genuinely needs them, don't put them in the prompt.
• Keep an ordering rule even for variable content. Retrieved material differs each time, but you can fix the ordering rule (say, by descending score), keeping at least the structure stable.
• Split long-lived rules out of the system prompt. If rules change frequently, put them in a separate section right after the system prompt — when they update, the system prompt part still hits. Prompt layering has direct financial meaning here.

Real-world effect

A typical support agent: system prompt + skill descriptions + fixed knowledge, about 1,500 tokens, with an average user question of 100 tokens. With a stable prefix, each call's input cost is roughly 1500 x cache price + 100 x full price; with no caching at all, it's 1600 x full price. If the cache price is about a tenth of the input price, input cost drops by over 80%.

For high-frequency, long-prompt workloads, this is the single biggest optimization — more direct than switching models or squeezing output.

Last point: track the cache hit rate as a metric. Most vendors show cache usage in the billing detail. Log it and compare month over month. Moving the hit rate from 0 to 80% may be your cheapest cost optimization.