Attention: words talking to each other
Queries, Keys, Values. Scores → softmax → weighted sum. Watch softmax(QKᵀ/√d)·V happen.
Before this
Next in How language models work
Multi-head attention + the MLPHow language models work, stop 7 of 202 minTranscript
It follows the voice. Click a sentence to jump there.
Take the phrase: a fluffy blue creature. On its own, creature is ambiguous — monster, pet, cartoon? What it means here depends on fluffy and blue. It needs a way to ask them.
Each token builds three small vectors from its embedding. A query: what am I looking for. A key: what do I contain. A value: what I'll hand over if anyone decides I'm worth listening to.
All four tokens do this in parallel. Then creature broadcasts its query, and the question is specific: are there any adjectives around here describing me?
That query gets dotted against every key in the sentence — one number per pair, measuring how well question matches answer. The adjectives score high. The article 'a' barely registers.
Raw scores come in whatever size they please — six point one, five point eight, no ceiling. Before they can act as weights, they need to behave like percentages.
Softmax squashes any set of scores into weights that sum to one. Fifty-five percent of the attention goes to fluffy, forty-one to blue, and scraps for everyone else.
Creature then collects the values of the tokens it attends to, scaled by those weights, and folds the sum into its own vector. It no longer means a generic creature — it encodes a fluffy, blue one.
Written down, the procedure fits in one line: softmax of Q times K transpose, divided by the square root of d, times V. Every symbol in it is a step you already understand.
And none of it happens one word at a time. Every token runs the full procedure at once, as a single matrix multiplication — the workload GPUs were built for. Attention is every word asking the others what it should mean here. Creature started ambiguous; it ends the sentence knowing it's fluffy and blue — because it got to ask.