Vectors and Similarity

Embeddings turn meaning into coordinates, so similar meanings land close. But similarity measures resemblance, not answerability — and that gap is where retrieval breaks.

Embeddings turn meaning into coordinates, so similar meanings land close together. That is only the first layer — a short distance says two sentences are about the same thing, not that one answers the other. Treating similar as relevant is the single largest error source in retrieval.

After reading, you should be able to answer:

• Why does retrieval find nothing when a user rephrases the question?
• Why do the most similar passages often fail to answer?
• Which two kinds of content can embeddings not read, and what covers for them?
• You ship a vector search: which metric do you look at first?

Literal matching misses any rewording

Start with the flaw in the old approach. Keyword search is literal matching: the user types certain words, and the system looks for documents containing them. Yet one meaning has many phrasings — the user types "my laptop will not boot," while the manual says "procedure for filing a repair on a device that fails to start." The user says "I'll push it to later," the document says "deferred." The strings share nothing, they mean the same thing, and matching on characters is helpless against all of them.

Embeddings fix exactly this. They bypass the wording and look at the meaning. Hand text to a dedicated embedding model and what comes back is a run of hundreds to thousands of numbers — a coordinate. Texts whose meanings sit close get coordinates that sit close too.

That map arranged by meaning is what you can picture as a semantic space: repair content packed into one neighbourhood, purchasing and expenses clustered in another, the company retreat somewhere far off. Retrieval puts the user's question onto the same map and asks which neighbourhood is nearest.

The embedding model that produces coordinates is a different kind of thing from the text models you know: a text model takes words in and hands words back out, while an embedding model takes words in and hands back a string of numbers. Distance between two coordinates is usually measured with cosine similarity. You do not need the math, only that it returns a number between 0 and 1 where a larger value means more alike. openai-embeddings

The nearest passage may still not answer

The most important point here starts with a small scene. A user asks, "Can I still use last year's unused vacation days?" Several passages come back, and the top two by similarity look like this: one is titled "Understanding Your Vacation Days," with the phrase packed densely throughout, and its body is nothing but a list of headings; the other covers how unused vacation is converted when someone leaves, again dense with the same phrase, answering a different question. The sentence that actually answers reads "unused days carry over until March 31 of the following year" — its wording barely overlaps the question, so it does not rank near the top.

Recall failures come in three broad shapes. The first is a hollow shell: a headline stuffed with the keyword and no substance behind it, yet the highest overlap with the question. The second is a neighbouring topic: a different question inside the same semantic neighbourhood, genuinely close in coordinate terms, answering something else.

The third is the hardest to spot, a stale version. The library still holds last year's guide alongside this year's, the wording nearly identical, so their coordinates nearly coincide and only a few numbers differ. A coordinate has no notion of time, and similarity alone cannot tell you which version is current. That has to be fixed at the data layer: retire the old document, or tag every passage with an effective date and filter on it during retrieval.

Similarity measures "how alike," not "does it answer." Whether a passage resembles the question depends on its wording and subject; whether it answers the question depends on whether the fact is inside it. These two often disagree, and both directions cause trouble:

• Alike but useless. An index page with keywords in every heading, a boilerplate notice repeated on every page — heavy overlap with the question, and they crowd out the real answer.
• Useful but unlike. The passage that answers may use entirely different wording: the user speaks casually, the document speaks in terms of art; the user uses the short name, the document the full one.

Why does this matter more than anything else here? Because every later stage of retrieval is built on this assumption. Taking the top few by similarity means assuming the most similar is the most useful. Once that assumption fails, the model holds the wrong material — and it will still answer confidently from it, looking even more credible because it cites a source.

The coordinates are learned, so they carry the corpus's biases

Nobody assigns these coordinates by hand. The model learns them from enormous amounts of text: words and phrases appearing in similar contexts end up with coordinates near each other. Put another way, they record how people use those words in the corpus — no objective truth involved — and that property becomes a problem later. word-embedding-wiki

The good news is that this layer costs very little. Among all model types, embedding models sit at the cheapest end, typically one or two orders of magnitude below text models. Turning your entire document set into vectors is barely a cost question at all.

What does need deciding up front is three engineering matters, because they determine whether this keeps working:

• It is a one-time offline step. Documents get split into chunks, each chunk gets embedded, and the vectors go into a store. At query time only the question gets embedded once, then distances are compared — which is why queries are fast.
• A changed document needs re-embedding. Once a document is edited, its vectors have to be produced again. This is easy to leave out of the plan, and the consequence is a knowledge base still answering from content that is months out of date.
• Switching embedding models means recomputing everything. Different models produce coordinate systems that do not transfer between each other; mixing them is like measuring distances on two different maps. Choose carefully, because a switch is expensive.

Negation and exact strings do not read; long text dilutes

Two clear blind spots. Put both on your design checklist.

One: it is insensitive to negation. "This operation requires two-person review" and "this operation requires no two-person review" use nearly identical words, so their coordinates sit close together. In compliance, policy, and medical content this is a real risk: it finds the passage with the most similar wording, not the one that reverses the meaning.

Two: exact strings are a weakness. Contract numbers, material batch numbers, employee IDs — these carry little semantics in a corpus, so as coordinates they pile up together, indistinguishable. And these are exactly what users ask about most.

One more thing that is not a blind spot but still matters: long text gets diluted. Squeezing a large block of text into a single coordinate amounts to averaging every topic it touches, and after the average none of them stands out. Chunking therefore decides retrieval quality directly, and the RAG article works through that.

Recall with it, do not rank with it

Turn the above into actions:

One: do not use similarity as the final ranking. It reliably removes the obviously unrelated, but it cannot separate "alike" from "useful." Recall a batch by similarity, then spend a more expensive step choosing the few that can actually answer.

Two: cover exact matching. Codes, batch numbers, and proper nouns belong to keyword retrieval, which judges them far more accurately than vectors do. Two ways to search, each blind is about wiring the two paths together.

Three: measure the retrieval layer first. Before touching the prompt, check whether the correct passage entered recall and at what rank. No model calls, fast and cheap, and it tells you whether the problem is finding or answering.

Embeddings turned "meaning" into a computable distance, and nothing replaces that — without it, a rephrasing finds nothing. But what they compute is only resemblance. Turning resemblance into usefulness takes mixing methods and reranking, which is the next layer.

Back to this chapter's thread: the model remembers nothing on its own. Everything it knows was placed there by your system in this one request. Whether the passages you placed are useful depends on your judgment at this step, not on the model's ability.

There is also a more fundamental problem than tuning similarity. In complex settings, the constraints that decide whether a judgment is right are often the ones worded unlike the question — a state machine that cannot skip states, an interface only one service may call. They score low on similarity and are still mandatory knowledge. Retrieval finds possibly relevant content; it cannot guarantee you gathered everything you needed, and that gap is the subject of Align the Domain Before RAG.