The whole Transformer, assembled
Embed → (attend + think) × 96 → unembed → probabilities. The box, opened; the loop, closed.
Next in How language models work
Mixture of Experts (MoE)How language models work, stop 9 of 202 minTranscript
It follows the voice. Click a sentence to jump there.
Each token becomes a column of numbers, and those columns pass through a stack of identical blocks. Every block does two things: attention, where the vectors compare notes, then an MLP — short for multi-layer perceptron — where each one processes alone. GPT-3 stacks ninety-six of these.
By the end of the stack, the last position's vector isn't about 'the' anymore — attention has folded the entire sentence into it. That one vector is all we need.
It gets scored against every word in the vocabulary, and softmax turns the scores into probabilities. Mat comes out around forty-six percent, floor at twenty-one — so mat it is.
Mat joins the input, and the machine runs again from the first block for the word after that, and the one after that. One box, one pass per word — every conversation you've had with a language model is this loop, repeated.
It compresses to almost nothing. Vectors, sharing, thinking, stacking, odds — everything a chatbot does is these five moves, run in a loop.
Nothing was left out. There is no hidden module where the understanding lives — just embeddings, attention, MLPs, and a score for every next word. Every sentence a language model has ever finished came out of this loop.