The problem attention solves
“River bank” and “bank deposit” start from the identical vector. Meaning must come from neighbors.
Next in How language models work
The only math you needHow language models work, stop 4 of 202 minTranscript
It follows the voice. Click a sentence to jump there.
I fished by the river bank. I deposit money at the bank. You resolved that ambiguity instantly — one bank is mud and reeds, the other is vaults and paperwork.
The tokenizer can't make that call. Bank is the same token in both sentences, so the lookup table hands back the same starting vector — every property identical, regardless of what surrounds it.
That leaves the raw embedding stranded between the river-edge meaning and the financial one, committed to neither. An average of two meanings isn't a meaning.
Context is the way out. Next to river and fished, the vector should drift toward the muddy, wet end of meaning space.
Next to deposit and money, the same vector should swing the other way — toward vaults, tellers, and interest rates.
So the fix is exactly what you did without noticing: let each word's vector be revised by its neighbors, pulled toward what its own sentence means. That mechanism is attention — the same bank token, dragged toward mud by river and toward vaults by deposit. A fixed embedding tells you what a word could mean; attention is how it settles on what it does mean here.