Hallucination

Every step it picks the most plausible next sentence, not the true one, and nothing checks the record. So you defend against it — you don't wait for a fix.

Fabrication is not a defect waiting on a patch; it is this machine's default behavior. Every step it picks the most plausible next sentence, not the true one. So the posture is defence, not waiting for a version that never invents.

After reading, you should be able to answer:

• Which kinds of content does it fabricate most convincingly, and need a gate?
• Why is what it invents often smoother than what is true?
• Which moves lower both the odds of fabrication and the cost when it happens?

The seven places fabrication clusters

Where it invents is not spread evenly; it clusters on a few kinds of content. Treat these as a design checklist: if your feature lands in one of these cells, it needs a gate, and the output cannot go straight to users.

1. Specific numbers, dates, version numbers. Strong format, weak semantics; writing a plausible value costs almost nothing.
2. Citations and references. It remembers patterns, not positions, so which paper, which page, has to be made up on the spot.
3. Interface names, field names, parameter defaults. Naming rules are rigid, so an invented one and a real one look identical on screen.
4. Obscure, niche, specialist topics. Few samples, flat distribution, anything can come out.
5. Anything past the cutoff. Never seen; it extrapolates from old patterns.
6. Your internal concepts and fields. Never in the corpus; it guesses from the literal wording.
7. The tail of long answers. Errors accumulate step by step, drifting further out.

The common thread: the more rigid the format and the less you can judge it by ear, the more you need an external check. An invented parameter name and the real one in the documentation are indistinguishable on screen; the only difference is that someone checked the docs.

It is not broken, it is working as designed

Back to the mechanism: at every step it picks one token from the candidates and appends it, and nothing checks the record. It only checks whether the sentence looks like it belongs in that position. So when a question lands somewhere it has never been, it does not pause either: there is always a highest-probability token waiting to be drawn, and after that one comes the next — a few dozen steps later, a lie with no grammatical flaw is complete. Where that mechanism comes from settles it: fluency and truth are independent, and it optimizes only the first. Wikipedia defines the phenomenon as presenting fabricated or misleading information as fact. hall-wiki

That list has the shape it does because of two habits layered on top of the mechanism. First, it has no "I don't know" state — in the weights, knowing and not knowing are not two signals that can be told apart; the only difference is whether the distribution is steep or flat, and when it is flat a word gets drawn all the same. Second, post-training amplifies confidence — the goal is a helpful answer, "I don't know" rarely collects reward, so supplying something always beats leaving the slot empty.

So it will not fix this on its own. The vendors' own guidance takes the same line: these techniques reduce fabrication a great deal but do not remove it. claude-hall What you can do is lower the odds and contain the cost.

Agree with your direction, and it invents harder

One trigger is not about content at all but about how you ask: put the conclusion inside the question.

"Is it because of X?" — and even when X is not the cause, it will likely run with X. In product terms: bring it a proposal and ask whether the proposal is better, and what comes back is confirmation rather than judgment. It follows your lead regardless of whether it understands — answers of that kind simply earn more reward during training.

That habit is most expensive in requirements debates and design reviews: you wanted objections and got agreement instead. To get around it, change the question from "is this good" to "what does it cost", or have it argue the opposing case outright — take the position out of the question.

Three lines of defence, each holding one stretch

Two things are available to you: lower the odds it happens, and contain what it costs when it does. In engineering terms they divide into three layers, each covering one stretch.

At the input layer, constrain the source. Do not let it answer from memory. Fetch the material the answer should rest on, put it into this request, and attach one hard requirement: answer only from the material above. This move does more than all the others, because the nature of the task changes — from "recall it from memory" to "read the material and find it," and the second is something it is clearly better at. How to fetch and assemble that material is the subject of RAG.

At the verification layer, demand citations and cross-validate. Have it mark which passage supports each conclusion: you can then check it, and annotating forces it to go back and confirm whether the claim has any grounding, which raises the cost of inventing. Run critical conclusions a second time, or have another model review them, and pull out anything the two disagree on for a human look; this is costly, so reserve it for cases where being wrong is expensive. A cheaper alternative: wherever code can do the check, let code do it — verify interface names against the docs, numbers against the database, formats against a rule.

At the product layer, make "I don't know" legitimate. State it plainly in the prompt: where the material contains nothing, answer "not mentioned in the material" and do not speculate. The interface has to make room for that answer too — an empty state, a handoff to a human, a prompt asking the user for more. If no "cannot answer" state exists in the interface at all, fabrication is the only path left open to the model.

Verification belongs in the system

Behind all three lines is one principle: the burden of verification belongs in your system, not pushed onto the user's vigilance. Showing users a line like "AI can be wrong, please verify" is the easiest thing to build and the least effective — it turns work the product owes into a chore for the user.

You do not need to stop every fabrication — only the ones that can hurt someone. Which judgments should never be handed over at all is the line drawn in It completes the implementation; it does not supply the judgment; the concrete moves are laid out in Three Moves to Check an Answer.

One test is more useful than asking whether this particular answer was right: would you find out if it were wrong? Where you would, you can let go. Where you would not, build the means of finding out before you talk about automating anything.