Where Training Data Comes From
What the corpus is made of decides whose bias the model carries and which parts it's missing.
The model's ability is not accidental; it is the shape of its corpus. The more text of a kind it saw, the better it is there; for the kinds it never saw, it can only invent.
Symptoms you'll recognize:
- It writes React fast and accurately, but its code for your internal framework is all fabrication.
- English technical questions get great answers; the same question in Chinese comes back shallower.
- It defaults to the most mainstream option and never thinks of the niche but better one.
What's in the corpus
Vendors do not publish their mixes, but the structure is clear: web pages dominate, with books and papers, code, and dialogue and Q&A as smaller blocks. On top sits a round of human-labeled preference data; that round decides how it talks, not what it knows.
Web pages give breadth: encyclopedias, forums, news, blogs, product pages. It knows a bit of everything and swallows the web's errors, stale claims, and marketing speak along with it. Its "industry best practice" is often just the phrasing that was written most often. Common Crawl's open web corpus is a typical source for this block. commoncrawl
Books and papers give long-range reasoning: coherent arguments spanning thousands of words exist only in long text. Copyright limits how much is actually available, less than you would imagine, which is why it is far more reliable on popular topics than on deep specialties. The Pile is one of the most careful public attempts to assemble diverse long-form text. pile
Code gives structural sense. It is not just that it can write code; text with strict syntax and machine-checkable correctness is extremely high-value training material. Its ability to emit valid JSON and fill formats faithfully comes largely from code.
Dialogue gives it the ability to talk like a person, and the habit of agreeing with you. Ask "is it like this?" and it tends to say yes. That is not understanding; that kind of response is simply rewarded more in the corpus and the labels.
Whose side it takes
The corpus is not a uniform sample of the world; it is a sample of "how much of this got written on the web." Written a lot, familiar. Written little, unfamiliar. The bias is systematic, and it lands in concrete places:
| Plenty in corpus | Little in corpus | What it means for you |
|---|---|---|
| English | Chinese and smaller languages | Same question, shallower in other languages |
| Popular frameworks, big-tech stacks | Niche tools, in-house builds | Its default recommendation may not fit |
| General-audience explanations | Real industry practice | Answers sound right, fall apart in practice |
| Success stories, positive narratives | Postmortems, negative detail | It is structurally optimistic about risk |
The last row matters especially in product work: ask it to evaluate a plan and you get the plan as advertised on the web, not as it works in real projects. Stanford's AI Index, which tracks model and data trends annually, shows the same pattern: the language and domain distribution of training data directly predicts where models perform well. stanford-ai-index
The response is plain: take the "which approach" decision out of its hands. Do not ask "which plan should we use"; tell it which plan to use and let it execute within that constraint. It executes the route you set; it is not good at choosing the route.
The missing block
One whole block has never entered the corpus: your company's internal documents, your business conventions, unpublished industry data, client lists, the process you changed last week. This is not "it doesn't know enough"; it is that the model has never seen them.
The danger is that its response to "never seen" is identical to "seen rarely": it gives you an answer anyway. Ask the meaning of an internal field and it will infer one from the name, in the same confident tone it uses for things it truly knows.
So you supply it, through exactly three paths: write it into the prompt, attach retrieval, or give a tool to look it up. Which one depends on size and how often it changes. A few hundred characters of fixed convention go into the prompt. Hundreds of pages that get updated go into retrieval. Data that changes every minute gets a tool.
Final judgment: if the correct answer depends entirely on information outside the corpus, the model should not answer directly. Your system should fetch the information and let the model process it. Miss this distinction and you have built a hallucination magnet.