Similar Is Not Relevant

The most similar passages aren't necessarily the answer to your question. This is the biggest error source in retrieval.

Vector retrieval scores every passage by similarity and feeds the top ones to the model. Sounds reasonable, but a fatal mismatch hides here: a high score does not mean the passage can answer.

Symptoms you'll recognize:

• The knowledge base holds the right answer, but the system cannot find it.
• It cites a source-looking passage, and the conclusion is backwards.
• Raising recall helps a bit, but cost and nonsense answers rise with it.

The gap between "similar" and "useful"

Vectors measure how close the wording and context are; words are mapped to a high-dimensional space and distances are measured between them. word-embedding-wiki Whether a passage can answer your question is a different thing, and the two only loosely correlate.

Look at the diagram. The user asks, "It's been seven days; can I still return it?" Three candidates:

First is a table of contents page. Its headers are full of "returns" and "FAQ," heavy word overlap, highest score. But it is titles with no body; there is not a single useful sentence.

Second is an exception clause from another category: "This category does not support 7-day no-reason returns." The wording nearly matches the question, so it scores high. But it says "not supported," and it belongs to a different category. The model may well answer "no."

The real answer is third. Take only the top two, and the correct passage never reaches the model. The model does not know that; it works with what it has and confidently produces a wrong answer.

This is the hardest knowledge-base bug to find: it cites the source, sounds well-founded, and is still wrong. Users trust it more precisely because a source is given.

Four typical false hits

Classifying the common cases makes debugging much faster.

Empty shells: tables of contents, indexes, navigation, title lists, dense keywords, no body.

Reverse clauses: "not supported," "excluded," "does not apply" — vectors are insensitive to negation, so the wording matches.

Neighboring topics: ask about refunds, get returns, after-sales, complaints — the same semantic neighborhood, genuinely close.

Outdated versions: old policies and historical documents use nearly identical wording, and vectors cannot tell new from old.

The last two are the most dangerous because the output looks perfectly reasonable; only someone who knows the business can spot the wrong citation. Outdated versions deserve special attention: vector space has no concept of time. A 2023 policy and a 2026 policy may sit at almost the same coordinates. Retrieval alone cannot separate them; you must handle it at the data layer, either retiring old documents or stamping each passage with an effective date and filtering at search time.

How to mitigate

You cannot eliminate this problem, but several measures help a lot, roughly in order of return. rag-failure-points

First, over-recall, then rerank. This is the standard fix. Recall 20 to 50 passages with vectors (rather too many than too few), then run a dedicated reranker that judges, passage by passage, "can this actually answer?" and keeps the best three to five. The reranker understands relevance better than vectors because it reads the question and the passage together instead of comparing separately computed coordinates. This is usually the single biggest RAG improvement for the cost.

Second, purge empty shells. Tables of contents, navigation, and index pages should be excluded at indexing time. Nearly free, and it eliminates an entire class of false hits.

Third, tag every passage with source and date. Store the source document, section, and effective date alongside each passage. Filter by time at search, and require the model to cite which passage it used. Once a citation is there, a business person can tell at a glance whether the quote is wrong; the problem flips from "undetectable" to "obviously wrong."

Fourth, set a similarity floor. If even the top score is low, the library has nothing relevant; have the system honestly say it does not know rather than build an answer from unrelated fragments. This blocks a good share of hallucinations.

Fifth, build a test set. As with prompt iteration: assemble real questions with their correct passages, and every time you change the retrieval strategy, run it and check whether the right passage makes the top few. Without this ruler, all tuning is guessing.