Align the Domain Before Reaching for RAG

Retrieval finds possibly-relevant material; it cannot guarantee the knowledge a complete judgment needs. Structure supplies coverage, and stale documents are the real risk.

Most teams building a knowledge base for an AI agent reach first for a vector store: dump the historical documents in, retrieve a few passages when needed, hand them to the model. Retrieval is useful, but it should not be the first move for a complex system.

The reason fits in one sentence: retrieval solves "find possibly-relevant material," and it does not thereby solve "does the model hold the knowledge set a complete engineering judgment requires." ai-agent-book

Retrieval cannot solve completeness

Put thousands of unstructured documents into one index and three problems arrive together. ali-architect-agent

Uneven granularity. A full design doc, meeting notes, an API note, a caption under a screenshot — any of them can become a retrieval unit, and their knowledge density varies enormously. Hitting a passage that looks relevant does not mean it covers the boundary that actually matters here.

Semantic similarity is not engineering relevance. Ask about "order cancellation" and retrieval will surface refund, fulfillment, marketing, and risk text that reads similarly — while missing constraints that decide whether the design is correct: this state machine must not be skipped, that message topic still has historical consumers, only the aggregation service may call this endpoint. Those constraints are not phrased like the question.

Top K gives no structural guarantee. The returned documents may come from different eras, different business scopes, and may contradict each other. The model gets fragments, not a complete chain. The context grows longer while attention spreads thinner.

So the problem with RAG is not "recall is too low." It is that RAG struggles to carry the job of knowledge modeling on its own. Retrieving across a mass of low-structure, low-density material tends to produce a document warehouse that is easier to search and just as messy.

A fixed structure supplies coverage constraints

Switching to a structured knowledge base, the first decision is the boundary. The answer is the domain: one knowledge base serves one stable business domain, without mixing internal rules across domains, and domains reference each other rather than duplicating.

A domain knowledge base needs at least five classes of content:

The structure is not for tidiness. It exists to give the agent a definite reading path: vocabulary first so concepts do not blur, then scenarios to see how business lands on the system, back to principles when consistency or compatibility is at stake, and to practice when the historical reason matters.

Put differently: a fixed structure supplies coverage constraints — it tells the agent what it must at least understand to solve a business problem. RAG only says that some material may be relevant. The former is a skeleton; the latter is an extension of retrieval. With no skeleton, faster retrieval still will not give the agent stable system understanding.

None of this means abandoning RAG. It is good at long-tail discovery, locating historical documents, open-ended questions, and supplying evidence. When the structured base lacks a detail, retrieve the source document and then write the confirmed conclusion back into the structured knowledge. Retrieval supplements; it is not the knowledge itself.

The real risk is a credible stale document

There is a risk in knowledge bases that gets underestimated: the danger is not missing documents. It is a document that has gone stale and still reads as authoritative.

A design written two years ago that was entirely correct then still reads smoothly today — still professional, still like a settled conclusion — while the system it describes may have changed long ago. The model cannot see "when was this last confirmed." It will weigh it exactly like everything else.

Maintenance therefore has to enter the process, with two complementary mechanisms.

Incremental maintenance. Tie knowledge updates to code changes: when code or configuration moves, a hook or CI job reads the diff, identifies the APIs, objects, databases, messages, and configuration knowledge likely affected, generates update candidates, and validates structural completeness. But do not swing to the other extreme and let any code change overwrite knowledge automatically. For high-risk knowledge — API contracts, database semantics, message schemas, state machines, historical compatibility, security policy — automation only discovers changes, generates candidates, and blocks omissions. A human still confirms the meaning. Tools hold the line that "the code changed, so the knowledge cannot stay exactly as it was." People decide what the change means.

Periodic calibration. Incremental work keeps things timely, but it does not reach cross-system, historical, and high-risk blind spots. Those tend to surface together at particular moments — a major sales event, a serious incident review, a systematic walkthrough. Treat the output of those moments as high-value calibration material; it closes structural gaps that daily increments cannot.

One more condition keeps a knowledge base alive: every important entry should show its source, last confirmed date, scope, current status, and owner. Without those, the base becomes another unmaintained document graveyard — except this time it actively misleads the model.

There is a harder instrument than scheduled reviews: let hit data decide what gets retired, instead of cleaning up on impression. Instrument the entries — use the coding tool's own hook to run a script after every file read or text search, check whether the target is the knowledge directory, extract entry anchors from the returned content, and accumulate hit counts plus last-read time (count an entry at most once in a short window; the script must exit silently on any error and never disturb normal reads). The resulting report ranks entries by hits and lists "zombie entries" never read at all. In periodic audits, review them one by one: delete what is stale, add keywords to what is useful but unfindable, expand and even promote what gets hit often. One detail is worth copying — that team deliberately excluded the knowledge layer from their code graph, forcing the agent to reach it only through file reads and text search, which is what makes the instrumentation fire at all tencent-800-rules.

Alongside it runs a one-way promotion chain: memory to document to rule, never straight to a rule. An experience verified repeatedly becomes a stable convention, gets promoted into a module document, and the original entry is deleted; a problem that keeps recurring shows memory failed to prevent it, and only then does it become a rule. Rules load every session and each new one dilutes the others, while memory is retrieved on demand and costs nothing when it misses. In one full audit, that team cut memory from 148 entries to 24 — most were not discarded but relocated — and of 13 evolution records, only 3 were ultimately promoted to rules. And the agent may only propose: evolution records land in a review queue, and a human decides each one tencent-800-rules.

Where the human moves to

What this approach ultimately changes is the human's role.

Engineers used to be the only retriever of system knowledge: search the repos, trace the call paths, write the design, hand the conclusion to the AI to implement. Now the research, retrieval, path reasoning, code verification, and design generation can be done independently by the agent. Human involvement shifts from "collect all the material for the AI" to "adjudicate the factual gaps and carry the key decisions."

That shift has a boundary that has to hold: when business trade-offs, cross-team interface commitments, data definitions, compliance requirements, or high-risk change authorization are at stake, the agent should stop guessing and raise an explicit question rather than filling the gap itself. An open-questions section in a design is not evidence of incompleteness. Quite the opposite: registering what existing material cannot confirm is what shows the design did not dress up inference as fact.

Two questions to check yourself: how many entries in your knowledge base can state when they were last confirmed? And when the model is uncertain, does it have a defined exit that hands the question back to a person?

The cost and the boundary

A structured knowledge base is not free. Three bills come with it. Upfront effort — extracting the tacit knowledge in an architect's head into structured documents is heavy human work. Maintenance — both mechanisms have to keep running, with someone confirming the meaning of high-risk changes. Applicability — it suits domains that are stable at the boundary and dense in knowledge; in an early stage where the business itself is shifting fast, a rigid structure becomes a burden instead.

To decide whether to build one, ask a single question: in your system, how many critical constraints are invisible in the code and known only to people who have been around? If the answer is "many," knowledge fragmentation is the main reason the agent behaves inconsistently, and the investment is worth it. If the answer is close to "almost none" — the system is simple, boundaries are clear, the code is the fact — then retrieval plus one good entry document is enough.