How the matrices get filled in
Weights start as noise. Guess, measure surprise, nudge downhill — trillions of times.
Before this
Next in How language models work
How the model knows word orderHow language models work, stop 13 of 202 minSee alsoThe gradient
Transcript
It follows the voice. Click a sentence to jump there.
Where do a hundred and seventy-five billion numbers come from? At the start, nowhere. Every weight begins as random noise, values drawn out of a hat — the model knows nothing.
Training starts with a quiz. Take real text, hide the next word, ask for a prediction. The truth is mat, and the untrained model gives it about a two percent chance — tied with purple. How surprised it was by the right answer becomes a number: the loss.
Gradient descent is the fix. Treat the loss as terrain — a hilly landscape over one weight's value. Find which direction is downhill, take one tiny step that way, and the surprise gets smaller. No cleverness — just rolling downhill until you can't anymore.
Backpropagation makes that affordable: a single backward pass computes the downhill direction for all hundred and seventy-five billion weights at once. Every weight learns how much of the error was its fault.
Repeat on trillions of words, and the noise organizes. Grammar first, then facts, then style. Mat goes from a long shot to the clear favorite, and the surprise falls from eight point four to zero point six — not because anyone taught it, but because guessing wrong got expensive.
So that's where the hundred and seventy-five billion numbers come from. Not from anyone writing them down — from guess, measure the surprise, nudge, a few trillion times over. The noise that knew nothing became every pattern the model has. No teacher, no rules handed in. Just being wrong, over and over, until being right was cheaper.