Keyword, Vector, and Hybrid Retrieval
Three ways to search, each with blind spots. Real systems almost always mix them.
"The knowledge base equals a vector database" is a widely repeated simplification. In practice, vectors have a clearly defined blind spot, and old-fashioned keyword search happens to cover it. This article is about how to combine them. The retrieve-then-generate route itself was defined by the RAG paper: take a passage from your material first, then let the model generate an answer. rag-paper
Symptoms you'll recognize:
- The user types an order number or product model; the knowledge base finds nothing.
- A paraphrase works, but the exact term fails.
- After you ship a new feature, users ask with the new name and recall fails entirely.
The two methods
Keyword search looks at literal overlap. Whatever the user types, it finds documents containing those words, scored by frequency and rarity. This is the approach search engines used for decades: mature, fast, explainable.
Its strength is exact strings: order numbers, product models, legal citations, names, error codes. These have no synonyms; the literal text is all the information.
Its weakness is paraphrase blindness. The user says "I don't want it," the document says "return," zero overlap, and it is missed.
Vector search looks at closeness of meaning. Its strengths and weaknesses are the mirror image: synonyms, casual phrasing, and related concepts are findable; exact strings are not. Why? An order number carries little semantic meaning in training data, so as a vector it sits among a crowd of other digit strings, indistinguishable. It is good at "close enough," and order numbers cannot be close enough.
Two more vector weaknesses are worth remembering: it is insensitive to negation, so "supported" and "not supported" sit close together, and it cannot handle new terms. A feature you named last month was not in the embedding model's training data, so the model has no idea where to place it. Keyword search has zero trouble with new words; literal matching is enough.
Why mix them
The two methods' strengths are complementary. Run only one and you have given up half your recall.
Hybrid search is simple: each path recalls a batch, then you merge, dedupe, and rank together. The engineering cost is small, and most vector databases and search engines ship the capability built in. Research on RAG failure modes reaches the same conclusion: single-method retrieval is among the most common failure points when queries are semantically similar but literally different. rag-failure-points
One detail when merging: scores from the two methods are not on the same scale, so you cannot add them directly. The common practice is to fuse by rank rather than by score, weighting each result by its position before merging. You do not need to know the algorithm, but you do need to know this is a tuning point; the default may not be optimal.
In practice, hybrid search is almost always better than single-method search, especially in enterprise knowledge bases that get both casual questions and exact codes.
Reranking is the key step
After recall comes a step many teams skip, and it is usually the highest-value work in the whole pipeline.
The idea: recall aims at not missing anything, so over-recruit, 30 to 50 passages across both paths. But you cannot stuff all of them into the context; it is expensive and dilutes the signal.
So add a reranker. It reads the question and each passage together and judges "can this passage answer this question," scores them again, and keeps the best three to five.
That is the key difference from vectors: vectors embed the question and document separately, then compare distances, so information is lost in compression. A reranker reads both together, so it judges far better. It is slower and pricier, but it only processes tens of passages, so total cost stays low. The "similar is not relevant" problem is mostly corrected at this step.
A starting configuration
Here is a recipe you can copy directly.
First, run two parallel recalls: vector and keyword, 20 to 30 passages each. Second, merge and dedupe. A passage found by both paths is likely truly relevant; give it a boost. Third, rerank and keep the top 3 to 5. Test the exact number; too few misses, too many dilutes and costs. Fourth, set a score floor. If even the top score is low, the library has no answer; take the "I don't know" branch instead of forcing the model to make something up.
One more rule, often ignored: look at how your users actually ask. Pull a hundred real query logs and you will quickly see whether your scenario leans casual or exact, and that tells you which way to weight the mix. This beats tweaking parameters, and most teams never do it.