Training vs. Inference

One feeds the model, the other uses it. The boundaries of cost and capability sit on either side of this line.

Training compresses a corpus into a set of weights; it happens once, and it is over. Inference runs those weights to compute an answer, and it happens every time you hit enter.

Symptoms you'll recognize:

• You correct a mistake; it acknowledges it, then repeats the same error in a fresh session.
• No cost pain in the demo, cost spiral after launch.
• Your boss asks "can we train our own model?" and you are not sure how to answer.

Two different things

Training: feed massive text, make the model guess the next token, adjust the weights when it is wrong, repeat trillions of times. Months, tens of thousands of GPUs, an astronomical electricity bill. What comes out is a weights file, and the file is then inert. The "bigger models and more data make better capability" curve, which the scaling-laws paper characterized, lives entirely on this side. scaling-laws

Inference: feed your input through those weights to produce output. Usually seconds, a fraction of one GPU's capacity, pennies. But it happens fresh on every call.

The most misunderstood point: nothing you say changes the weights. When you correct it in chat and it listens this round, that is because your correction entered this call's input sequence and the next token could reference it. Close the session and that input is gone; the weights are still the same file.

You can only touch one side

Draw this line and many arguments end.

On the training side, your only move is selection. Knowledge cutoff, language strength, code ability, native temperament — all fixed at training. Not satisfied? Switch models. There is no second path; no prompt will make a model that is weak in Chinese good at Chinese.

On the inference side, you can change a lot: how the prompt is written, how much context, what temperature, whether to attach retrieval, whether to give tools, which model tier, whether caching is on. Day-to-day AI product work is 90% here.

In between sits fine-tuning: a small extra training round on existing weights that changes style, format, and domain framing, not knowledge. It costs orders of magnitude less than training but far more than prompting, and it hardens once done — less flexible than a prompt. openai-finetune Exhaust prompting and retrieval before considering it.

Where the cost comes from

Inference bills per token, input and output priced separately, output usually several times more. The bill is simply: call count times input length times input price, plus output. Of the three multipliers, the one you lose control of is the middle.

The classic spiral: the prompt starts at 200 characters and works well. System rules grow it to 800. Then history, retrieved documents, and tool results join, and one request is now 20,000 tokens. Every click costs you 20,000 tokens, and every turn resends all that came before.

What you can squeeze: cache the unchanged prefix, summarize long documents before they enter context, route simple tasks to a cheaper tier, compress multi-turn conversations periodically. All on the inference side, all yours to do.

One thing people forget: latency is an inference-side cost too. The longer the output, the longer the wait, because tokens stream one at a time. For a snappy product, the most effective lever is usually making the model say less, not upgrading to a stronger model.