Why Hallucinations Happen

Hallucination is an inevitable byproduct of the generation mechanism, not a bug. You can only defend, not fix.

Hallucination is not a bug; it's the direct product of the generation mechanism. Its job is to continue with the most likely next sentence, not the true one. Most of the time those coincide. Sometimes they don't.

Signs you'll recognize:

• It names a function that doesn't exist, with parameters that look perfect and aren't in any documentation.
• It cites a paper—author, year, journal all present—that turns out to be invented.
• It gets ninety percent right and the remaining ten percent is wildly wrong, delivered in the exact same tone as the correct parts.

Why it's inevitable

Back to the mechanism: at every step it picks one token from the candidates and appends it. Nothing in that process checks whether the sentence is true. It checks whether the sentence looks like it belongs in that position.

So when it's asked about something it has never seen, it doesn't stop. There's always a most-probable token to draw, then another. A few dozen steps later, a perfectly grammatical lie is finished. Fluency and factual accuracy are two independent things, and it optimizes only the first. Wikipedia's article on AI hallucination defines it as responses that present fabricated or misleading information as fact. hall-wiki

Four mechanism-level details are worth remembering separately:

• It has no "I don't know" state. Knowing and not knowing aren't two distinguishable signals in the weights—just a sharper or flatter probability distribution. When the distribution is flat, it still draws a word.
• Patterned content is easiest to fabricate. API names, parameter formats, paper citations, statute numbers, URLs—rigidly formatted content is effortless to invent and especially convincing.
• Post-training amplifies confidence. It's trained to be helpful, and saying "I don't know" isn't usually rewarded, so it tends to give an answer anyway.
• It follows your lead. Ask "is it because X?" and even when it isn't, it will likely expand along X. Loaded questions are hallucination hotspots.

So there's no day when this gets "fixed." What you can do is lower the probability and contain the damage afterward—not wait for a hallucination-free model. Anthropic's official guide says outright that these techniques reduce hallucinations significantly but don't eliminate them. claude-hall

Where it shows up most

Treat this table as a design checklist: if your feature lands in any of these cells, it needs a validation step—don't hand the output straight to users.

Four defenses

One: constrain the source. Don't let it answer from memory. Pull out the material the answer should be based on, put it into this request, and explicitly require "answer only from the material above." This single move outranks all others because it turns recall into reading comprehension—a task the model is far stronger at.

Two: demand citations. Ask it to mark which passage of the material supports each conclusion. Two benefits: you can verify, and the act of annotating forces it to check whether each claim has grounding, raising the cost of fabrication. Beware the inverse trap: any source it volunteers that isn't in your material should be treated as fake.

Three: cross-validate. Run critical conclusions twice, or have a second model judge them, and pull the disagreements out for human review. It's expensive, so reserve it for places where being wrong is costly. The cheaper version: validate with code where possible—look up API names in the docs, check numbers against the database, validate JSON against a schema.

Four: make "I don't know" legitimate. State explicitly in the prompt: if the material doesn't cover it, answer "not mentioned in the material" and don't speculate. And make room for that answer in the product—empty states, handoff to a human, a prompt for more info. If your interface has no "can't answer" state, the model has no choice but to fabricate.

The last point is an attitude, not an action: put the burden of verification in your system, not in the user's vigilance. "AI may be wrong, verify yourself" is the laziest and least effective warning. What actually works is constraining scope, keeping provenance, and locking down formats so errors are caught before they reach the user.