Context Window

The window is a budget, not a bucket. System prompt, history, retrieved documents, tool output, and answer headroom all compete for it, and overflow drops things silently.

Chapter 2 ended on model selection. This one takes up a more basic question: your own business, documents, and internal concepts are not in its corpus — so where does it get what it doesn't know? First stop, the capacity ceiling. The context window is the upper bound on everything one request can hold, counted in tokens. It is a budget, not a bucket — more things compete for it than most people assume, and when it overflows it will not tell you what it dropped. context-window-wiki

After reading, you should be able to answer:

What competes for the same space inside one request?
What are the two overflow outcomes, and which is worse?
Why does a bigger window not solve "just put everything in"?
Where does the truncation rule belong, and why must it be explicit?

One request has five claimants

Most people assume the window holds "the back-and-forth conversation." In practice one request has to carry five things at once:

System prompt. The product's setup and rules, resent every round, roughly fixed in length and position.
Conversation history. All prior exchanges, growing continuously.
Retrieved documents. The passages retrieval feeds in — often the largest block.
Tool output. In agent scenarios every tool return piles in, and this is the growth source that most easily runs away.
Answer headroom. The block most people forget.

Separate them and you notice the first four are all input, yet behave nothing alike: the system prompt is a constant, history grows monotonically, retrieved documents depend on how many you chose to include, and tool output depends on how many steps the agent has taken. Only one of the four is something you actively control each round; the rest grow on their own.

Headroom for the answer comes out first

That last item deserves unpacking: output takes up window space as well. Push the input right up against the ceiling and barely any room is left for writing an answer. So the budget should be built in the reverse order — set aside enough for the output first, and whatever remains is what the input can use.

One more thing trips people up: the advertised context length counts tokens, not characters. For the same number, Chinese text usually fits somewhat more than you would guess and code usually less — the only accurate count comes from measuring. So "how much room is left" cannot be eyeballed; it needs a measured conversion.

When it does not fit, the default is to drop the start

The outcome comes in two forms, and which one you get depends on the platform and on your own implementation.

The request errors out. This is the better of the two — the problem is out in the open and you can handle it.

The far more awkward one: the opening content is trimmed away without a word. Plenty of frameworks and products cut the front of the request automatically so that it can go through at all.

The awkward part is that the front is often exactly what mattered most — the rules written at the start, the original description of the task, the background the user first gave. Lose those and the model's behavior begins to drift, with nothing shown to you at all. All you feel is "it seems a bit dim today," and you cannot trace why.

So there is a practical discipline: first find out what your framework actually does on overflow. Does it raise an error, or cut something away? If it cuts, which end? Is any of that recorded? The answer settles one thing — whether you will have any thread to pull when this goes wrong later.

The point to hold on to: what gets dropped is decided by your truncation rule, not by the model. If you do not write a rule, the framework writes a default one for you — and a default rule generally does not distinguish "important" from "verbose." So the rule has to be explicit, and it has to answer three things: which end gets cut, how much is kept, and whether it is recorded.

A bigger window only moves the wall further out

With million-token windows everywhere, the conclusion people draw is that "put everything in" settles the matter. It does not. Three reasons why.

One: fitting is not reading. Once content runs long, the middle gets passed over noticeably — where content sits decides how well it gets read. Feed in a whole manual and the opening and the close may be the only parts doing real work. lost-in-middle

Two: it is expensive. Every passage in the context is sent again and billed again on every round. A long document chatted over ten rounds multiplies the input volume by ten. Long-window bills climb quickly, and inside an agent-style loop it shows up fastest.

Three: it is slow. The longer the input, the longer the wait before the first token appears. Users notice this directly.

So the idea of "the window is big, put it all in" gets refused three times over: the model may never read it, you pay for it every round if it does, and the user waits longer first. The reverse is what holds: include only the passages that genuinely earn their place — which is the problem RAG exists to solve. Retrieval does more than find material; it also keeps the irrelevant out and leaves only the relevant passages in.

Subtract in this order, then log what you used

Subtract in order, and the order itself carries meaning:

Subtract answer headroom first, estimated from the longest answer you permit, with a little margin on top. Then the system prompt, a fixed value you compute once. Then a typical amount of tool output — if you are building an agent, this block often exceeds the documents. Only what is left gets split between history and retrieved documents.

Then set a few rules:

Cap retrieved documents hard. Cap both passage count and total length; counting passages alone is not enough, since passages differ in size.
Compact history past a threshold. How to compact and what it drops is the compaction article.
Make tool output truncatable. Converting structured results into a summary before inserting them saves far more than piling in the raw payload.

One last item: log how much was actually used. The data is worth a lot — which feature costs the most, which way usage is trending, how much room is left before the ceiling. Most teams only notice any of this once production starts erroring, and by then it is too late.

The window is only the container. How you arrange what is inside it is a second layer of design: the same content is used with different probability depending on where it sits. That is what Where Content Sits in the Window takes on. And the budget itself has more sources competing for it than the five named here — What Actually Goes Into Context breaks down the full list, each with its own retention rule.