Three Checks on an AI Answer
No need to verify line by line; three moves filter out most fabrication.
The model's ability to fabricate is severely underrated: it makes things up fluently, confidently, and with citations. Verifying sentence by sentence is not realistic, but three moves filter out most of the errors that would hurt you.
Symptoms you'll recognize:
- It cites reports, authors, and URLs that do not exist.
- It lays out arguments that sound right but die at the first counterexample.
- Percentages, dates, and amounts do not add up, and you almost used them.
The three moves
Check the source. For every citation, figure, name, and paper in the answer, open the original. This filters out the most fabrication: the model is best at inventing plausible-looking sources, from report titles to author names. If the citation cannot be found, the claim is basically untrustworthy. When you do find the original, compare its numbers against what the model quoted; it often "remembers" roughly and writes it down.
Find a counterexample. For every "always," "definitely," and "never," think of one case where it fails. It says "this method definitely lifts conversion"; find a scenario where it does not. If it names exceptions and states its boundaries, it actually understands; if it insists there are no exceptions, it probably did not think it through, or it is just agreeing with you. This move targets the model's sycophancy: it tends to produce the conclusion you expect to hear.
Recompute key numbers. Percentages, amounts, dates, conversions — do the math yourself. This is the densest fabrication zone and the easiest to catch: when the number does not check out, that sentence is out, no matter how smooth the rest is. Thirty seconds of arithmetic stops the most dangerous class of errors.
Why three is enough
The three moves are not random; they map to the three zones where models fabricate most.
Sources are easiest to invent. Producing a credible-looking citation costs the model nothing, so checking sources is the highest-value move. hallucination-survey
Logic is the easiest to flatter. The model is trained to give reasonable, smooth answers and rolls on even when unsure; its goal is not to be right but to look right. Counterexamples are the cure for that smoothness.
Numbers are the easiest to get wrong. Token prediction is inherently bad at exact arithmetic; it learned "text roughly like this," not real calculation. Recomputation is the killer move.
Together the three cover fabricated sources, agreeable logic, and wrong arithmetic, the main forms of fabrication. For whatever scattered small errors remain, weigh them with one question: "could this hurt me?" If yes, check; if not, let it go. The anti-hallucination guidance from major providers lands on the same line: identify high-risk cases, then verify and constrain deliberately. claude-hall
When it's mandatory
Not every answer needs all three moves. Grade by scenario.
Just browsing: skip them. Inspiration, brainstorming, understanding a concept — being wrong is harmless, so take it at face value.
Forwarding or reporting: at least check sources. You are accountable for the content; verify it first.
Decisions or operations: do all three. Money, contracts, data changes — this is the class you cannot skip. The model recommends; the decision and the responsibility are yours.
And the final point: make verification a product feature, not a personal habit. If your product shows answers that could influence a user's judgment, design in sources, verifiable links, even a "this passage may need verification" hint. Answers with citations do not just improve engineering hygiene; they move "is it trustworthy" from the user's problem to the product's design.