What is temperature in LLMs?
Set it to zero expecting the same answer twice, and you will still catch the model changing its mind on the third run.
Temperature
Sampling temperature
Temperature is a number, usually between 0 and 2, that controls how much randomness a language model uses when choosing its next word. A low setting favours the most likely word every time. A high one lets weaker candidates win more often.
A model does not pick one next word. It scores every word in its vocabulary and gets a ranked list of candidates. Temperature reshapes that list before a word is drawn from it.
A low value sharpens the gap between the top candidate and the rest, so the same word tends to win. A high value flattens the gap, so a weaker candidate gets picked more often. That is the whole mechanism: one number, reshaping a list of odds.
Why the setting matters in practice
A support assistant answering a policy question wants the same answer every time a customer asks the same thing. A brainstorming tool wants a different idea on each run. Temperature is the one setting that separates those two jobs.
Teams reach for temperature 0 to make a system feel predictable, especially anywhere the output feeds a test suite or a compliance record. That instinct is right. The mistake is trusting it completely, because temperature 0 narrows the odds without deleting them.
- ScoreEvery candidate word gets ranked
- ReshapeTemperature sharpens or flattens the gap
- 0.0Near-greedy, still not fixed
- 0.7Varied, useful phrasing
- 1.5Loose, sometimes incoherent
The same prompt run three times at each setting will not return three identical answers, even at 0.0.
01What temperature should I use for an AI assistant?
For a support or policy assistant, use a low temperature, around 0 to 0.3, so answers stay consistent. Drafting, brainstorming or varied copy works better at 0.7 to 1.0, which gives useful range without turning incoherent. Above about 1.2, output starts breaking down for most tasks.
02Does temperature 0 make an LLM deterministic?
No. It narrows the model toward its single most likely word at each step, so output varies far less. But batching and floating-point rounding on the provider's hardware can still change the result between runs. Treat it as consistent, not identical.
03What is the difference between temperature and top-p?
They reshape the odds in different ways. Temperature sharpens or flattens the model's favourite word. Top-p, also called nucleus sampling, works on the list itself. It trims the candidates down to the smallest group that adds up to a probability threshold, then samples from that shortened list. Many APIs let you set both, and changing one shifts what the other does.
