The Context Window

Everything a single request can hold — that's the size of the table you get.

The context window is the upper bound on everything a model can see in a single request. It's measured in tokens, and — here's the catch — it holds more than most people assume. context-window-wiki

Symptoms you'll recognize:

• By round thirty, it starts forgetting the rules set at the beginning.
• You dump a whole document at it and the answers get vaguer.
• Some long inputs error out for no obvious reason; short ones don't.

What's on the table

Most people think the window holds "the conversation." In reality it must hold five things at once. Picture a fixed-size table: every item you place on it shrinks the room left for everything else.

• System prompt. The product's setup and rules, resent every round.
• Conversation history. All previous exchanges, including tool results. This keeps growing.
• Retrieved material. Document chunks fed in by RAG. Often the single largest block.
• The user's current message. This round's question and attachments.
• Output headroom. The block most people forget.

That last one deserves unpacking. Output also consumes the window. If a model advertises 128K of context and you stuff the input to 127K, there's almost no room left to write an answer. So when budgeting, reserve output headroom first; only what remains is available for input.

Another common misconception: "200K context" means 200,000 tokens, not 200,000 characters. Chinese text fits a bit more than you'd estimate, code often less — the only accurate count comes from measuring.

What happens when it overflows

Two outcomes, depending on the platform and your implementation.

• The request errors out. This is actually the good case — at least you know something's wrong and can handle it.
• The more troublesome one: the earliest history is silently dropped. Many frameworks and products trim the front of the conversation automatically so the request can go through.

The problem: the front of the conversation is often exactly what matters most — the rules set at the start, the original task description, the background the user gave. Once dropped, the model's behavior starts drifting, with no warning at all. You just feel "it's a bit dumb today" and can't find the reason.

So there's a practical discipline: find out exactly what your framework does on overflow. Does it error or trim? Which end does it cut? Is there a log? The answer determines whether you can ever debug it later.

Does a big window solve it

With million-token windows all over the place, people assume "stuff everything in" is the answer. It isn't. Three reasons:

• Fits doesn't mean readable. When content gets long, the middle gets noticeably ignored. Stuff in a hundred pages and it may actually use only the beginning and the end. lost-in-middle
• It's expensive. The whole context is re-sent and rebilled every round. A 50K-token document, chatted over ten rounds, is 500K tokens of input. Long-window bills grow fast — especially in agent-style multi-round loops.
• It's slow. Longer input means more noticeable time-to-first-token. Users feel it.

So the right mindset isn't "big window, stuff everything" but "only include the few passages that are actually needed" — that's exactly what RAG exists for. Retrieval isn't just about finding material; it's about putting in only the relevant passages.

Budgeting the window

Treat the window as a budget to allocate, not a bucket to fill as full as you like.

One allocation scheme you can use directly: subtract output headroom first (estimate your longest allowed answer, plus a margin); subtract the system prompt next (a fixed value, computed once); split what's left between conversation history and retrieved material.

Then set two rules: a hard cap on retrieved material (say, at most 5 chunks of 500 tokens each), and compress the history once it passes a threshold.

Finally, log actual token usage. This data is extremely useful: you can see which feature costs the most, how it trends, and when you're about to hit the ceiling. Most teams only discover this when production starts erroring — by then it's late.