Temperature and Randomness

It samples from a distribution at every step, and that sampling is deliberate. Turn it down where there is one right answer, up for divergence — down means consistent, not correct.

Ask the same question twice and get two different answers, and the first instinct is that something is broken. The opposite is true: drawing one sample from a distribution at every step is how it was built to work, not a defect waiting on a patch. That single mechanism governs two things at once — the variety you get, and the variance you have to defend against.

After reading, you should be able to answer:

What is wrong with "turn it down and it gets more accurate"?
Why does the same question come back different, and why is longer output less stable?
Which tasks should have randomness pinned to the floor, and which should keep their range?

Narrow it and you get the same answer, not a better one

The most common misreading of this randomness is worth putting first: narrowing it does not make it right.

"Turn the randomness down and the answers get more accurate" — in that sentence, "accurate" is standing in for "correct." What actually happens: with the distribution sharpened, every draw lands on the same most-likely option, so you see the same answer, not a better one.

If that high-probability option is itself wrong, narrowing the randomness only makes you reliably wrong in the same way. In production that is harder to catch than random error — random error draws attention; consistent error gets treated as fact.

So keep two things separate:

For consistency, narrow the randomness. Batch processing and reproducible runs are right to pin it to the floor.
For correctness, supply material, supply checkable evidence, and keep the final verification step. Randomness cannot help here.

Two boundaries belong alongside that. First, the floor does not make the output well-formed: anything feeding a downstream system still has to pass a structure check, with a retry or a fallback when it fails — randomness is an objective fact of the system, so design with it in mind and do not expect to dial it away like a bug. Second, low randomness makes output formulaic: the same phrasing, the same shape, tiresome by the third read. If what you are producing is copy a person reads, the floor is often the wrong call — that case is precisely the one that wants several different versions.

It does not take the best one; it draws one

To make sense of that, go back to what it does as it writes each word: the candidate table at each position is computed, and then a token is drawn from that table rather than taken from the top — Guessing the next token covers that motion in full.

One point belongs here because it is the only thing this dial touches: the shape of the candidate table is the only thing temperature changes.

So error compounds. Emitting a few hundred tokens means drawing a few hundred times; one draw that lands elsewhere and every distribution after it shifts. That is why longer output is less stable — nothing to do with it getting muddled later on; the accumulated draws simply pile up.

One point deserves stating: even with randomness at its floor, the same prompt is not guaranteed to come back word for word. Concurrency during batching, the order of floating-point operations, a server-side version change — any of these produces small differences. The floor means "nearly identical," nowhere near "cryptographically identical." Genuine consistency means caching the result yourself and reusing it.

Temperature decides how bold it is about an unusual word

Everything above is dialled with one parameter, and its name is temperature. There is usually a second dial alongside it.

Temperature governs "how bold is it about reaching for an unusual word." Turn it down and the field of choice shrinks to the handful of most confident candidates: conservative, repetitive, safe. Turn it up and long-tail candidates get lifted, so the phrasing is fresher — and the odds of drifting rise with it. gpt3-paper On this one dial, creativity and reliability sit at opposite ends; you cannot have both at once.
There is also a "draw a line first" dial. It ranks candidates by probability, accumulates from the top, and stops once the configured share is reached, discarding everything past the line. Its value: however flat the distribution, anything that obviously should not appear is out of reach. In a product it makes a better safety valve than temperature.
Not every API hands you these dials. Some newer reasoning models do not take a temperature at all, or take one that has no effect. Confirm this at selection time, or you will build a plan on a dial that cannot be turned. Vendor text-generation docs define these parameters precisely. openai-textgen

One right answer, or not, decides how much range it gets

There is one test: does this task have a single right answer? If yes, pin randomness to the floor. If no, leave some range. By that test, everyday calls fall into three bands.

One right answer. Pull the signing date, the two parties, and the amount out of a contract; attach a topic label to each piece of user feedback; reshape free text into a record with fixed fields; turn a sentence of business description into a query. There is exactly one correct outcome, so any variance is an error. Pin it to the floor.

Accuracy with room for wording. Translate an English note into Chinese, rewrite a casual request as a formal specification, compress a meeting transcript into a summary: the facts and the meaning cannot drift, but which word gets used can vary a little. Low is enough; do not push it to the floor.

Divergence is the point. Ten headline candidates for one product, three versions of ad copy in three different registers, a few angles on how this feature could be cut — this band wants difference, so leave the range wide open. The only thing to guard against is it wandering off into invention.

A product usually runs several kinds of calls at once, and they should not share one configuration. Put field extraction and headline writing on the same setting and neither comes out well.

Before you reach for this dial, the prompt has work to do

When someone reports "unstable output," the first suspect is not this dial but a prompt that is too loose. No format, no examples, no statement of what is forbidden — those cases make up the bulk of it.

Fill in the four parts, add examples, pin down the output format, and the stability you buy back dwarfs nudging one number down a notch. There is an order behind this: fill in whatever parts are missing, add the examples, get the techniques right, and leave parameters until all of that is done. Parameters sit last not because they are useless, but because they paper over the fewest problems — they make the same answer show up again; they do not make that answer more correct.

With the earlier steps done, the last job is judging whether this particular adjustment helped — the ruler is still that one fixed sample set: the ruler comes before the dial.