PMaker home
Search
中EN
System
Color theme
空格的键盘
空格的键盘空格的键盘产品经理 · AI Builder
When the answer is wrong, don't touch the prompt first — walk forward in this order, and most problems sit in the first two stagesIs it in the index?Search the index directly and pull up the raw passageMany "wrong answers" are content that was never writtenWas it recalled?Log every passage this query retrievedThe right passage may never have entered recallWhat rank?Rank 18 with top-5 taken means it wasn't recalled at allThat's a reranking problemRight material, wrong answer?Paste the passages and the question into a chat manuallyOnly this failure points at the promptTouching the prompt when the first three stages failed is pure waste

"The knowledge base answers wrong" isn't one problem — it's four. Pin down which stage, and the fix becomes obvious.

Where RAG Goes Wrong

Chunking, recall, and reranking — every stage can fail, each with its own symptoms.

Once your knowledge base is live, you'll get "it answered wrong" reports. The most common reaction is to edit the prompt — and most of the time, the problem isn't there.

Symptoms you'll recognize:

  • You keep tuning the prompt and nothing moves.
  • The same question is right sometimes, wrong other times, with no clear pattern.
  • The business side says "it's right there in this document," and it still can't answer.

Walk forward through the chain

RAG is a serial pipeline; break any link and the final output is wrong. So diagnose front to back, not from whatever is easiest to tweak. The paper that systematically catalogs RAG failure modes lists them in pipeline order — the order itself is the diagnostic path. rag-failure-points

For one wrong case, ask these four questions in sequence:

1. Is the information in the index at all? Search the vector database directly with relevant keywords and pull up the raw passages. You'll be surprised how many "wrong answers" are cases where the document never said it — or the info lives in a file that was never indexed. This one step filters out a meaningful share of tickets.

While you're here, check: does the sentence exist completely in one chunk, or was it split across two by chunking?

2. Was it recalled? Print every passage this retrieval returned (this requires you to have logging in place — see below). Is the correct passage in there?

3. What rank? If it was recalled but sits at rank 18 while you take the top 5, it effectively wasn't recalled. That's a reranking problem.

4. Right material, wrong answer? Paste the retrieved passages and the question into a plain chat window and see what the model says. Only if it fails here do you touch the prompt.

This order matters. If a failure lives in the first three stages, editing the prompt is wasted effort — the model never had the right material, and no amount of prompt craft fixes that.

Fixing each stage

1. Missing from the index → content problem, not technology. Either the document never said it (go write it), the document wasn't ingested (go add the source), or it exists only as images and scans with no extracted text. Fixes here usually aren't engineering. The real bottleneck in many knowledge-base projects is that nobody wants to clean up the documents — say this at project kickoff, not after launch.

2. Not recalled → retrieval strategy problem. Common causes and fixes: the user's words are too far from the document's words (add a keyword-retrieval leg); exact strings like order numbers and model numbers (vectors are useless here; you need keyword); chunking dropped the heading so the passage has no topic signal (prepend the section path at index time); or you retrieve too few (raise the count — better to over-recall and let reranking filter).

3. Ranked too low → add reranking, or tune it. If you don't have reranking yet, this is usually the highest ROI improvement you can make. If you do, check whether a high-similarity decoy — a table of contents page, a reciprocal clause — is crowding out the right passage.

4. Right material, wrong answer → now it's prompt and model. Typical causes: you didn't say "answer only from the material," so it mixes in its own knowledge; there's no "I don't know" escape, so it invents when material runs short; the material is too long and the key passage sits buried in the middle — an effect that a dedicated paper has quantified; or the question genuinely needs cross-passage reasoning beyond the model's ability. The lost-in-the-middle effect is documented and measurable. lost-in-middle

Build an evaluation

The walkthrough above handles individual cases. For systematic improvement, you need a repeatable measure.

The minimal setup: a set of "question + correct document passage" pairs, 30 to 50 of them, covering common phrasings and a few nasty edge cases.

Then read metrics on two layers:

Retrieval layer — did the right passage enter recall, and at what rank? No model calls, fast and cheap, run it frequently. Use it to judge chunking changes, embedding-model swaps, and recall-size tweaks.

Answer layer — is the final answer correct? This needs human or model-assisted judgment, slow and pricey; run it on bigger changes.

The reason to keep them separate: if the retrieval layer score won't climb, the answer layer can't get better. Fix retrieval first — that's the effective order.

Finally, two engineering prerequisites before launch:

1. Log every retrieval's results and ranks. Without this, the diagnostic above can't even start — you won't know what was recalled.

2. Make answers carry citations that open the source. Then the business side can judge citations themselves, and feedback changes from "it answered wrong" to "it cited passage 3, but should have used passage 5" — the latter is directly actionable.

References

  1. Seven Failure Points When Engineering a Retrieval Augmented Generation System
  2. Lost in the Middle: How Language Models Use Long Contexts