Debugging RAG

Editing the prompt is the easiest reaction and usually the wrong one. Walk the stages in order — in the index, recalled, ranked, or answered wrong anyway — and the fix follows.

Once a knowledge base is live you will get "it answered wrong" reports. The most common reaction is to edit the prompt — and most of the time the problem is not there. Retrieval is a serial chain, so diagnose it in chain order rather than starting wherever editing is easiest. rag-failure-points

After reading, you should be able to answer:

• Why is prompt tuning so often wasted effort?
• Walking the chain forward, what does each step rule out?
• How do the fixes for the first three stages differ from the fix for the fourth?
• How do you turn "it was right today" into a repeatable metric?

Do not start at the last link

The prompt sits at the very end of the chain. Starting there means none of the earlier suspects have been ruled out.

Put one wrong case on the table and ask these four questions in order; each one rules out a batch of possibilities.

One: is the information in the store at all? Search it directly with relevant keywords and pull the raw passage up. A good share of "wrong answers" turn out to be cases where the document never said it — or it sits in a file that was never ingested. Getting this far filters out a meaningful share of tickets. One thing to confirm while you are there: does that sentence exist completely inside a single passage, or did chunking divide it across two so each half is only partial?

Two: was it recalled? Print out every passage this retrieval hit. Is the correct passage among them?

Three: what rank did it get? If it was recalled but sits far down the list and you only take the top few, the practical effect is the same as not recalling it.

Four: right material, wrong answer? Paste the recalled passages and the question into a chat by hand and see whether the model gets it right. Only a failure here makes prompt editing worthwhile.

The value of this order: when the failure lies in the first three stages, editing the prompt achieves nothing — the model never had the right material, and no amount of prompt polish compensates. Conversely, if all four checks pass, the chain itself is sound and you need to look somewhere else.

The first three failures are never a prompt problem

The first three stages map to three different diseases with three different cures — and they share one property: editing the prompt does nothing for any of them.

One: missing from the store → a content problem, not a technology problem. Three shapes: the document simply never said it (go write it), the file exists but was never ingested (go add the source), or the material is images and scans whose text has not been extracted. Fixes here usually are not engineering. What stalls a lot of knowledge-base projects is that nobody is willing to spend the time cleaning the documents up first — that should be stated at project kickoff, not discovered after launch.

Two: not recalled → a retrieval strategy problem. Common causes and fixes:

• The user's words are too far from the document's words (add a keyword leg).
• The query is a string that has to match character for character, like a contract number or a device serial. Vectors are useless there, you need keywords.
• Chunking dropped the heading, so the passage carries no topic signal. Prepend the section path at index time.
• The recall count is set too low. Raise it first — better to fetch more than needed and let reranking filter.

Three: ranked too low → add reranking, or tune it. If you do not have reranking yet, this is usually the highest-return improvement available. If you do, check whether a high-similarity decoy — a copyright page, a revision history — is crowding out the right passage. Their wording often sits closer to the question than the actual answer does.

Only the fourth failure is yours to fix in the prompt

Work through the first three and the material is confirmed to be in the store, recalled, and ranked high — yet the problem survives. Only now is it about the prompt and the model, and the usual slips are these.

Nothing said the answer may draw only on the supplied material, so it mixes in what it already knows; no room was left for "I don't know," so it fabricates when material runs short; the material is long and the key passage sits buried mid-context; or the question itself calls for reasoning across several passages, beyond what the model can do. The mid-context effect has a dedicated paper quantifying it. lost-in-middle

This diagnosis belongs last because its evidence is the weakest: only when the correct material is placed in front of the model and it still answers wrong can you be sure the fault is on its side.

Two things that have to exist before launch

The walkthrough above only works if the chain leaves traces behind. Two of those traces have to be in place before launch, because neither can be added retroactively: the moment a problem appears, that request's context is already gone.

One: write every retrieval's hits and ranks to a log. Without that record, not a single step of the walkthrough above can be taken — there is no way to know what was recalled.

Two: answers must carry citations, and a citation must open the source text. Then the business side can check the citation themselves. And the quality of the feedback changes entirely: instead of a vague "it answered wrong," you get "the passage it cited is the supplementary clause; the main clause is in another section" — and that kind of report can be acted on directly.

Their value also reaches past debugging — recall logs and openable citations are what let you tell "never found it" apart from "found it and still answered wrong," and therefore optimize those two problems separately.

Past single cases, you need a repeatable yardstick

The walkthrough above handles single cases. Systematic improvement needs a repeatable yardstick.

The minimal version: a batch of "question + correct passage" pairs, covering the common phrasings plus a few nasty edge cases.

Then read two layers:

Retrieval layer — did the correct passage enter recall, and where did it rank? No model calls here, so it is fast and cheap, which means you can run it often. Chunking changes, embedding-model swaps, and recall-size tweaks all get judged by it.

Answer layer — is the final answer correct? This one needs a human verdict, or a model verdict with spot checks, and it is slow and pricey, so run it only on bigger changes.

Keeping them apart matters because if the retrieval layer will not climb, the answer layer cannot get better. Getting retrieval right first is the effective order. And the set itself ages as the product changes — question phrasings drift, documents change — so those pairs need ongoing care, which is the point of Tend the Eval Set Like a Product.

If you have checked every stage, fixed what was broken, and it still answers wrong, the question becomes a different one: is retrieval even the right tool for this? Adding knowledge is almost always retrieval's job, but some needs were never knowledge problems in the first place — to add knowledge it is almost always RAG draws that line.