What a language model actually does
One job: guess the next word. Probabilities over every word, sample, append, repeat.
Before this
Nothing. This is a starting point: it needs no other lesson first.
Next in How language models work
Tokens and embeddingsHow language models work, stop 2 of 201 minSee alsoDistributions
Transcript
It follows the voice. Click a sentence to jump there.
A language model has one job: given the text so far, guess the next word. The cat sat on the — and it has to fill in that blank.
The answer isn't a word. It's a probability for every word the model knows. Mat, forty-six percent. Floor, twenty-one. Couch, nine. Moon, point three — a strange place for a cat.
Sample one of the front-runners — mat — and stick it on the end. The sentence is now one word longer.
Feed the longer sentence back in and the model produces a fresh set of probabilities. Maybe a period ends it. Maybe 'and' keeps it going. Same move: predict, add, repeat.
Every reply you've gotten from ChatGPT or Claude is this loop — one word chosen, glued on, and fed back in, thousands of times.
So a language model is one box guessing the next word, over and over. That box has a name: the Transformer. No magic, no understanding, no plan — just the cat sat on the, one word at a time, all the way to every answer you've ever read.