Temperaturehow models choose
Greedy loops and drones. Temperature sharpens or flattens; top-p cuts the weird tail.
Before this
Next in How language models work
Scaling lawsHow language models work, stop 19 of 201 minSee alsoSampling
Transcript
It follows the voice. Click a sentence to jump there.
The Transformer's final output isn't a word — it's odds. For 'the cat sat on the', mat gets forty-six percent, floor and couch trail behind, and moon barely registers. The network's work ends there; picking a word is a separate job.
Always taking the favorite has a failure mode: the cat sat on the mat, the cat sat on the mat, forever. Greedy decoding gets stuck in loops, because the most likely word is usually the least interesting one.
Temperature reshapes the odds before the roll. At zero point two, nearly all the probability piles onto mat. Cold means safe, focused, predictable.
At one point five, the same set of probabilities — the distribution — flattens out and even moon gets a real slice. Hot means surprising — occasionally the cat sits on the moon.
Top-p keeps the smallest set of words that covers about ninety percent of the probability. Mat and floor make the cut. The tail gets thrown away, and what's left is re-divided. Moon can't sneak in at any temperature.
So the network stops at odds, and the choosing is yours. Temperature and top-p are that choice made into two numbers — sharpen for safe, flatten for surprising. The creativity slider in every AI app is exactly this pair, reshaping the odds before every roll.