Temperature and Randomness
Getting a different answer each time is by design, not a bug. When to turn it off.
It is not a mood; every step is a probability draw. That draw is designed in, and you can turn it off.
Symptoms you'll recognize:
- The same prompt run twice: once with perfect formatting, once with an extra stray sentence.
- It works in the demo and breaks in front of the boss.
- Batch processing a thousand items; thirty come back malformed, and there is no pattern.
Where randomness enters
For every token, the model first computes a full table: the probability of every token in the vocabulary. Up to this point everything is deterministic; the same input produces the same distribution. gpt3-paper
Randomness enters at the next step: drawing one from the table. Not taking the maximum, drawing. The token at 62% is likely, the one at 18% still has its chance. Once a non-obvious token wins, the rest of the sentence grows in that new direction.
So the variance amplifies. Generating a few hundred tokens is a few hundred draws, and one different pick shifts everything that follows. That is why longer outputs are less stable.
One clarification: even with randomness fully off, the same prompt is not guaranteed to reproduce byte for byte. Concurrency in batch processing, floating-point ordering, server-side version updates — all introduce tiny differences. Zero temperature means "almost identical," not "cryptographically identical." For true consistency, cache your own results.
The two dials
Temperature is "how bold with rare words." Low, it picks only from its most confident candidates: conservative, repetitive, safe. High, the long tail lifts, wording gets fresher, and the chance of nonsense rises with it. Creativity and reliability are the two ends of one dial; there is no free lunch.
Top-p is "draw a cutoff line." It ranks candidates by probability, adds them up to your chosen fraction, and drops everything after. The benefit: no matter how flat the distribution, you never sample something that clearly should not appear. As a safety valve in products, it is often more useful than temperature.
Not every API gives you these dials. Some newer reasoning models ignore temperature or do not accept it at all. Confirm during model selection; do not build a plan on a parameter you cannot turn. OpenAI's text-generation docs define both parameters and give sensible value ranges. openai-textgen
When to set it to 0
The judgment has one line: does this task have a single right answer? If yes, set it to 0; if no, keep a little randomness. A quick cheat sheet:
| Scenario | Suggested | Why |
|---|---|---|
| Field extraction, classification | 0 | One right answer; any variance is an error |
| Structured JSON output | 0 | The format must be identical every time |
| Code, SQL generation | 0–0.2 | Working beats beautiful |
| Translation, rewriting, summarization | 0.3–0.5 | Needs accuracy, allows some wording freedom |
| Copy, headlines | 0.7–1.0 | You want several different versions |
| Brainstorming | 1.0+ | The wild ones are the point |
A product usually has several kinds of calls, and they should not share one parameter. Running field extraction and headline generation at the same temperature makes both worse.
Two practical notes. First, don't rush to tune temperature. Most "unstable output" is actually a loose prompt: no format, no examples, no prohibitions. Fix those and stability improves far more than moving from 0.7 to 0.3.
Second, stability ultimately needs validation. Temperature 0 makes the same answer more likely; it does not make it correct or well-formed. Anything that enters a downstream system should be format-validated, with retry or fallback on failure. Design for randomness as a fact of life, not as a bug you can tune away.