Knowledge Cutoff

The corpus decides whose bias it carries and which gaps it has — and collection stopped on a date. Anything that can go stale has to come from you.

This chapter asks one question: which of its answers can you trust? The root of the answer sits in one place. Its ability is not accidental; it is the shape of its corpus. And that corpus has two built-in gaps: it only holds what was written down in public, and collection stopped on a date. So whole regions of the world never appeared in it at all.

After reading, you should be able to answer:

Why does it answer with total confidence about your internal conventions and get them wrong?
Why does the same question come back at different depths in another language, or about another stack?
Why are "latest version" and "current price" the questions it answers worst?

What never entered the corpus does not exist to it

Your company's internal documents, your own definitions and conventions, unpublished industry data, client lists, the process you changed last week — none of it appears anywhere in the public corpus. This is not "it does not know enough"; it is that the model has never once seen it.

The danger: "never seen" and "rarely seen" produce the same response in it — it answers anyway. Ask the meaning of a field only your company uses and it will guess from the literal name, in exactly the tone it uses for things it genuinely knows. That is where fabrication concentrates — not because it refuses to say "I don't know", but because it has no such state.

Whatever the corpus holds, it is strong at

Vendors do not publish their mixes, but the broad shape is clear: web pages are the largest block, then books and papers, then code, plus dialogue and Q&A. On top sits a round of human-labeled preference data, which carries no facts — only posture.

Web pages give breadth. Encyclopedias, forums, news, blogs, product pages — so it can join almost any conversation; the price is that the web's errors, stale claims and marketing copy went in with it. Its "industry best practice" is often nothing more than whichever phrasing got repeated the most. commoncrawl
Books and papers give long-range reasoning. Arguments that stay coherent across thousands of words, building step by step, exist only in long-form text. Copyright limits how much is actually available, less than outsiders assume — which is why it is far more dependable on popular topics than in a deep specialty. pile
Code gives structural sense. Text with strict syntax, where a machine can rule on correctness outright, is unusually valuable training material. Its ability to produce structured output a parser accepts on the first try owes a great deal to code.
Dialogue gives it a human voice. Natural phrasing and the rhythm of question and answer are learned from this block.

One round of data carries no knowledge, only posture

The preference round answers a different question: which replies people like better — the tone, the length, whether the conclusion comes first. It shapes posture, and has nothing to do with how many facts are on board. The same body of knowledge can produce a product that speaks plainly or one that keeps circling.

One side effect deserves its own paragraph: it follows your lead. The tendency has nothing to do with how many facts are on board — it is purely what the reward produced. Answers that run with the questioner's presupposition collect more reward in the labels. Ask a model for its opinion through that kind of question and what you get back is confirmation, not judgment. How that trigger works, and how to get around it, is covered separately in Fabrication.

The category written about most is the one it knows

The corpus is not a uniform sample of the world; it is a sample of how much of each thing got written down. Written a lot, familiar. Written little, unfamiliar. The bias is systematic, and at least these layers come out of it:

Language. Ask the same question in English, then in Chinese, and you get different depths; the smaller the language, the wider the gap.
Stack. The approach the community discusses most is on the tip of its tongue; the layer your own team wrapped around it, it can only guess from the name.
Register. Explanations written for a general audience are everywhere; how the work is actually done inside the industry rarely gets written down — so its answers sound right and fall apart in the project.
Narrative direction. Success stories and promotional copy are abundant; postmortems and negative detail are scarce. So it is structurally optimistic about risk.

That last layer matters especially in product work: ask it to evaluate a plan and what you get back is the plan as advertised on the web, not the plan as it runs in real projects. The language and domain distribution of the training data is a direct predictor of where a model performs well. stanford-ai-index

The response is plain: take the "which approach" decision out of its hands. Do not ask it which plan to use; state the plan and let it work inside that constraint. Executing the route you set is what it is good at — choosing the route is not.

Past that day, it extrapolates from old patterns

Collection ends on a date. Training, evaluation and internal testing follow, and only then does the model ship. So the model you use is far older than today, and usually months older than its own release date. Vendors state roughly where that date falls in their model docs; look it up, but do not read it as a precise boundary. claude-models

More important than the date itself: in practice the line is not clean. Different sources have very different recency — a standard document may go years without revision while a leaderboard moves every week — and the closer you get to the cutoff, the thinner the online discussion and the thinner the data, so that stretch is foggier to it than much earlier periods. The accurate statement is not "it knows everything before that month"; it is the newer the fact, the less you can rely on it. One detail is easy to miss: ask when its own knowledge stops and the answer is generated too; for an accurate figure, read the vendor docs.

Past that line, its behavior is consistent. It has no memory of what came after, but it has seen plenty of patterns for things of that kind, so it follows the pattern. The result is highly plausible and simply not true. Four situations catch people out most often:

Fast-moving content with a rigid format. Model numbers, batch codes, standard identifiers, administrative codes — the format pins the shape of the answer, so filling in a value that does not exist is easy and looks flawless.
Relative time. "How many days are left in the quarter", "how far off is that deadline" — it does not know today's date, so the starting point is wrong and every step after it is wrong too.
Anything phrased as "latest" or "current". The moment those words appear in a prompt, settle one question first: does it know when now is?
Leaderboards and benchmark scores. What it reports is the figures from whichever version entered the corpus; who leads and by how much has long since moved, yet it tells you as if it had just checked.

Anything "latest" is yours to feed

The fix is not on the model's side; it is on yours: anything perishable has to be handed over in this request, and you cannot count on it being remembered.

Put the date in the prompt. One line giving today's date fixes every relative-time calculation, at almost no cost.
Retrieve before you ask for facts. Pull the real text into the context and require the answer to be based on it. That converts a recall question into a reading-comprehension question, and accuracy changes qualitatively.
Send live data through tools. Flight status, desk occupancy, account balances — anything that changes every minute is beyond what retrieval can keep up with, and the model has to look it up on the spot.
Pin the version. When it calls a dependency, put the version number in the prompt and hand over the signatures of the key functions, so it does not guess which release it is talking to.
State which source wins. Say it explicitly: where the material you supplied differs from its prior knowledge, the material wins. Without that line it blends the new facts and the old ones into a version that is wrong on both sides.

Which path to take depends on two properties of the knowledge: how big it is, and how often it moves. A convention that fits on one page and does not change for months is cheapest in the prompt; a whole manual, revised every few weeks, belongs in retrieval; something that changes by the minute can only go behind a tool that looks it up on demand.

One layer up, there is a move in the interface too: label the information with its time. If the answer came from model memory, say that it reflects knowledge up to a date; if it came from live retrieval, show the source and the time it was fetched. When users can see the timestamp, misreading costs much less.

That yields a bottom line: if a scenario's correct answer lies entirely outside the corpus, it should not be handed to the model to answer directly — let the system fetch the information first, and let the model do the processing. Draw that line badly and you have built a fabrication magnet. The standard way to hand outside knowledge in at question time is RAG.