Fine-Tuning and RAG

Fine-tuning teaches how to say it; RAG supplies what to say. Trying to install knowledge by fine-tuning is the most expensive misjudgment in the field — you find out months later.

Of all the forks in an AI project, this one carries the highest price when you get it wrong — and it usually takes months to surface. Yet in the overwhelming majority of cases there is only one answer: to add knowledge, use retrieval, not fine-tuning. rag-wiki

After reading, you should be able to answer:

• What one question separates the two?
• Why can fine-tuning not install facts, and what does it actually learn?
• On the single job of adding knowledge, what three extra costs does fine-tuning carry?
• When the model is not good enough, what order do you try things in?

One question decides it

Fine-tuning operates on the model itself. You assemble a batch of input-output samples and let it keep learning; the internal weights shift, and its output preferences shift with them.

RAG operates on the material handed to it. Not a single weight changes. The only difference appears at query time, when the relevant documents are placed inside that one request.

So the first question is always: is it missing knowledge, or is its delivery off?

It cannot state how many working days you promise for a response — missing knowledge, so reach for RAG. It can recite the commitment but rambles and sounds nothing like support — a delivery problem, so start with prompting and only then talk about fine-tuning.

Remember it in one line: fine-tuning governs how it talks; retrieval decides what it has to talk about.

Training hammers phrasing, not the list of facts

The conclusion runs against intuition, so it is worth a few extra sentences: training it on your own documents sounds like the most natural thing in the world.

What training gives a model, though, is a grasp of statistical regularities and habits of phrasing. Facts do not get stored the way rows get written into a table. Feed in several hundred ticket records and what comes out is "text like this looks a certain way, favors particular vocabulary, tends to unfold in these structures" — while the individual contents, like "an overdue ticket escalates within four hours," are not in there at all.

Hence the textbook failure: once training finishes, its tone does carry the flavor of your documents, yet ask about a specific clause and it invents one anyway — and the invention reads as more plausible because the register is right. That is a worse position than before: the wrong answer now wears a credible disguise.

On knowledge alone, fine-tuning carries three more bills

What was just described is a matter of mechanism. What follows is bookkeeping, and all three entries point the same way: any need that means "make the model know something" is cheaper to serve with retrieval.

One: the source cannot be traced. This material has dissolved into the weights, so no sentence can be traced back anywhere. RAG, by contrast, can supply citations — often a condition that simply has to be met in an enterprise.

Two: changes take effect slowly. When one policy changes, samples get rebuilt, training gets rerun, evaluation gets repeated. On the RAG side you swap in a new copy of the document and re-index, and that is the end of it.

Three: existing abilities may be weakened. Train only on narrow-domain material and performance elsewhere can drop. Catching that requires a full evaluation, and plenty of teams never run one.

Where fine-tuning does belong

With all that said, fine-tuning's own value comes out sharper: its fit is well defined — every case lands on form, none on facts.

One: you need an output format or style that never wobbles. Prompting can hold most calls, but if you need it to hold across hundreds of thousands without drifting once, it has to be trained in.

Two: the domain has its own conventions of expression. Tender documents and inspection reports each follow a fixed form: describing them in a prompt stretches it out enormously, and every call has to carry that description again.

Three: you want to shrink the bill for a long prompt. This is a real account to settle: when the prompt is long and calls are frequent, training those rules into the model means later requests no longer have to bring them along.

Four: you want small-model cost with big-model results. Train a small model on one narrow task and it may come close to a large model's quality at a fraction of the spend. For repetitive, single-purpose work the arithmetic works out.

Put the four together: not one of them aims at "teach it new facts."

Choose this path and the price belongs in the same calculation: high-quality samples have to be produced in quantity (producing them is a project in itself), training costs money, and an eval set has to exist to confirm nothing regressed — and the moment a vendor ships a new version, all of that may need doing over. The vendors' own model-optimization documentation does not put fine-tuning first either; it sits inside a workflow of "evaluate → prompt → fine-tune → re-evaluate." openai-finetune

When the model falls short, try things in this order

Whenever the model is not good enough, walk down this sequence — most teams find the problem gone by the second rung.

Rung one: fix the prompt. Least spend, fastest result, reversible at any moment. Fill in the parts a prompt ought to have, add a few examples, and the majority of problems end right here.

Rung two: add RAG. If knowledge is what is missing, there is only one direction.

Rung three: move to a stronger model. When reasoning is the ceiling, swapping models is far less trouble than fine-tuning, and possibly cheaper.

Rung four: only now does fine-tuning qualify — and first state plainly which specific problem the three rungs above could not solve. If you cannot state it, do not do it.

One closing note: the two are not mutually exclusive. Well-built products often run a fine-tuned model together with RAG — fine-tuning pins down output style and format, RAG supplies facts that are current and correct. They were never the same problem, so layering them is natural.

What this chapter leaves you with

This chapter was nine views of one thing: the model remembers nothing on its own; everything it "knows" was placed there by your system in this one request.

First, find the right passages to put in context. Similar sits close, but closest is not useful shows similarity is not relevance; three ways to search, each blind shows you have to mix them and then rerank. Then ask whether the container holds it. One request holds this much is a budget; the ends read clearly shows position is itself a weight; long chats drop things shows anything that must survive cannot live only in the conversation; remembering has a cost shows cross-session memory is the same store-and-fetch pipeline.

Finally, join the two halves: retrieve then answer is the skeleton of the pipeline, and when it answers wrong, walking the checks forward localizes the broken stage.

Together they give one judgment: this layer is far more under your control than the model layer. Pick the wrong model and you can only swap it out, but retrieval, chunking, ranking, budget, and position are all code you wrote — each one changeable, testable, and reversible. So "it answers wrong" is usually not a model problem. It is an engineering problem.

The next step is the system layer. Retrieval can only guarantee "find possibly relevant content," never "we gathered everything that matters" — in complex settings the constraints that decide whether a judgment is right are often worded unlike the question and will not surface on their own in a retrieval result. So the next move on knowledge is not to tune retrieval better; it is to put the knowledge structure in place first: Align the Domain Before RAG.

That closes the knowledge line: fitting, finding, and wiring. Next comes a more concrete bill — what one call actually costs. It starts with the fact that it does not charge by the character.