Training and Inference

Training happens once and sets the ceiling. Inference happens on every call and is where all your money goes — so it is where everything you can change lives.

A model gets fed in the factory long before it gets used in your product, and those two phases differ in what they cost, what you can change, and how often they happen. Everyday conversation tends to collapse them into one question — "how good is the model?" — and that collapse is where product decisions start going crooked.

After reading, you should be able to answer:

• Why does a correction you make in chat not survive into the next session?
• What can prompts, retrieval, and fine-tuning each actually change?
• Why does the bill creep out of control after launch?

The weights that are fixed at the factory

The feeding phase is training: massive text goes in, the model guesses the next token over and over, and every miss adjusts the parameters. This is the vendor's job, run continuously for months across thousands of chips at an electricity bill no product budget should carry. What ships at the end is a weights file, set for good — it does not change because of something someone told it yesterday, and it does not teach itself new tricks. The "bigger models and more data make better capability" curve describes this phase. scaling-laws

The using phase is inference: your input goes through those weights and comes out as text, seconds later, at a cost measured in fractions of a cent. The crucial part is that every call starts from zero. Nothing the previous call computed leaves a trace, and nothing gets cheaper because you already asked — change a single word and the whole thing is computed again from scratch.

Hold on to one line and most of the surrounding arguments resolve themselves: the ceiling is set at training, and the bill recurs at inference.

Close the session and what you said is gone

Here is the counter-intuitive part: it registers your correction and still does not write it into the weights.

The correction takes effect immediately because that passage of yours was placed into this call's input sequence, where the token being generated could read it. But that input lives only inside this session. Testing it is easy: ask the same wrong thing again in a fresh session and the original mistake comes back in full. Your correction was never "remembered" — it merely happened to sit in the same passage of text as you.

Keeping the two apart matters. "It remembered" means your system stuffed the content back into the next input; "it learned" can only come from training. In product terms, every "make it remember" design — system prompt, long-term memory, retrieved documents, tool results — is the same action implemented differently: load the material again on the next call. Which is why what actually goes into context deserves more of your time than how smart the model is.

Training leaves you selection; inference leaves you dials

Draw this line and a lot of arguments end on the spot.

On the training side, your only move is swapping the model. Which day the knowledge stops at, which language it handles well, whether it can code, whether it can read images, what its native temperament is — all locked in at training time. If it does not suit you, change your selection; there is no other route. No prompt makes a model that cannot read images read one. Choosing a model per task is the whole of the work on this side.

On the inference side, you have plenty to move. How you phrase the prompt, how much context you give it, which temperature you pick, whether retrieval is attached, whether tools are offered, which tier of model you choose, whether the cache is on — every one of those dials is yours. Day-to-day AI product work spends nine tenths of its effort here.

So here is the test: when it "can't do something," first work out which side the problem is on. A selection problem means switching models; almost everything else still has room.

Fine-tuning sits in between: it shifts tendencies, not knowledge

Between training and inference stands another layer, fine-tuning. It runs one more training round on the existing weights using a small amount of data, and it too produces a weights file that is then fixed. What it adjusts are tendencies, not how much it knows — style, format, and domain framing can move; specific entries like "what our company policy says" cannot be installed this way. It costs orders of magnitude less than training but far more than editing a prompt, and once done it is locked in, with far less flexibility than a prompt. openai-finetune

So when is it worth it? When four conditions hold at once: the output format has to be extremely stable, the framing has to be strictly uniform, the prompt has already become unwritable or unstable, and you have a few thousand decent examples on hand. Meet all four and what fine-tuning buys is not having to repeat the same instructions in every prompt.

Go the other way — you want it to know something that happened last month, or a return policy you just changed, or you have a few dozen examples — and the answer is nearly always retrieval: find the material and put it into the context. Fine-tuning or RAG draws a finer line.

Do not reverse the order: exhaust prompts and retrieval before considering fine-tuning.

How the bill grows, and four places to squeeze it

Inference bills per token, with input and output counted separately, and the output portion usually costs several times more. So one request costs "input length times input price" plus "output length times output price," and multiply by the calls in a day to get your bill. Stack those factors together and the per-request input length is the one that slips away from you. how that unit works covers the units themselves.

The swelling usually grows like this. A support assistant starts with a prompt that says little more than "reply to the customer politely" — a few dozen words. To make the answers more expert, a section of the product manual gets attached to context, so a request starts at three thousand tokens. Then the full ticket history, the knowledge-base passages that matched, and tool results all get wired in, and the same question now ships forty thousand tokens into the model. The customer clicks once, you pay for forty thousand tokens, and the next turn resends everything from the turn before on top of it.

Four places to squeeze, all on the inference side, all yours to decide:

1. Pin the unchanging prefix so it can be cached.
2. Route simple tasks to a cheaper tier.
3. Summarize long documents before they enter context.
4. Compact a long conversation every so often.

Ranked by how little they disturb product logic, the first two barely touch anything and are worth doing immediately; the last two change process design and pay back more.

One more bill that people forget: latency belongs to the inference side too. The longer the output, the longer the user waits, because the content streams out one token at a time. For a responsive product, reaching for a stronger model is usually not the most effective move — making it say less is.

And a closing caution: watching only the total hides the problem. A bill that went up might mean the feature got more complex, or that failures and retries got more frequent — and those two have completely different fixes. Attribute Cost Down to Every Step is about making the number locatable.