Fine-tuning or RAG

To add knowledge, it's almost always RAG. When fine-tuning is actually the right call.

This is the most expensive fork in an AI project. Teams that pick wrong usually take months to notice—and in the vast majority of cases, the right answer is the same one.

Signs you'll recognize:

• Your boss says "let's fine-tune our own model" and you're not sure whether to push back.
• The design doc says "fine-tune on company documents," which sounds entirely reasonable.
• After fine-tuning, it still doesn't know the company's rules—and now it's also starting to invent things.

The core difference

Fine-tuning changes the model itself. You take a batch of input-output samples and continue training, adjusting the weights so the output tendencies shift.

RAG changes the material it sees. The model isn't touched at all; relevant material is simply placed in front of it at query time. rag-wiki

One line to remember: fine-tuning teaches "how to say it"; RAG supplies "what to say."

So the first question is always: is it missing knowledge, or is its delivery off?

If it doesn't know the company's refund policy, that's missing knowledge—use RAG. If it knows the policy but answers verbosely and doesn't sound like a support rep, that's a delivery problem—try prompting first, then consider fine-tuning.

Why fine-tuning can't add knowledge

Worth unpacking, because it's counterintuitive: "train it on our documents" sounds like common sense.

The problem is that training learns statistical patterns and ways of expressing things—it doesn't file facts into a table. Fine-tune on a thousand company policies and the model learns "text like this is formal, with certain recurring structures." It does not learn "the return period is seven days."

The result is the classic failure: after fine-tuning, it sounds like your documents, but when you ask about a specific clause it still fabricates—and the fabrication is more convincing because the tone matches. That's more dangerous than before.

Three side effects follow:

One: no provenance. What goes into fine-tuning dissolves into the weights; you can't trace any sentence back to a source. RAG can cite its references, which is often a hard requirement in enterprise settings.

Two: updates are slow. A policy changes and you must re-prepare samples, retrain, and re-evaluate. RAG means replacing the document and re-indexing—minutes.

Three: existing abilities can degrade. Fine-tuning on narrow data can lower performance elsewhere. Finding that requires a full evaluation, which many teams never run.

When fine-tuning is the right call

Fine-tuning isn't useless; it has clear use cases—and they all concern form, not facts.

One: you need an extremely stable output format or style. Prompting gets you 80-90 percent there, but you need 99 percent, held constant across a massive number of calls.

Two: there's a specialized domain register. Medical, legal, or financial document conventions are painful to describe in a prompt.

Three: you want to cut long-prompt cost. If your prompt is thousands of tokens and call volume is huge, baking the rules into the model means not resending them every call. The savings can be substantial.

Four: you want small-model performance approaching a big model. Fine-tuning a small model on a narrow task can approach a large model's quality at a fraction of the cost—great for high-frequency, single-purpose tasks.

Note the common thread: none of these is "teach it new facts."

Also count the cost: thousands of high-quality samples (preparing them is a project itself), training spend, an eval set for regression, and the risk of redoing everything when the provider ships a new version. OpenAI's model optimization docs place fine-tuning inside an "evaluate → prompt → fine-tune → re-evaluate" workflow, not as the first thing you reach for. openai-finetune

The right order

When the model isn't good enough, walk this sequence. Most teams stop at step three.

Step one: fix the prompt. Cheapest, fastest, instantly reversible. Round out the four prompt parts and add a few examples; most problems end here.

Step two: add RAG. If knowledge is missing, this is the only correct direction.

Step three: switch to a stronger model. If reasoning is the bottleneck, swapping models is far simpler than fine-tuning and may cost less.

Step four: only then consider fine-tuning—and you must first name the specific problem that steps one through three couldn't solve. If you can't answer, don't do it.

One final note: it's not either-or. Mature products often run "fine-tuned model + RAG"—fine-tuning pins the style and format, RAG supplies current, accurate facts. They solve two different problems and stack naturally.