How the model knows word order
Attention is order-blind. Position fingerprints and RoPE put the sequence back.
Before this
Next in How language models work
No peeking: causal maskingHow language models work, stop 14 of 201 minThis opens up
Transcript
It follows the voice. Click a sentence to jump there.
Dog bites man is a Tuesday. Man bites dog is a headline. To everything we've covered so far, they're indistinguishable — the same three tokens in a different order.
The culprit is attention. It compares vectors and never asks where a vector came from. Position one, position nine — no difference. Both sentences collapse into the same unordered pile: bites, dog, man. A bag of words.
The classic fix happens before any of the blocks. Every position gets its own vector — one for slot one, another for slot two — and you add it to the word's embedding. The sum carries both facts: this is dog, and dog came first.
Those position vectors come from sine waves — several of them, each cycling at its own frequency along the sentence. One drifts slowly, one faster, one faster still.
Sample all of them at position four and you get a combination of heights no other position can reproduce. That combination is the fingerprint.
Models like Llama skip the addition. Rotary position embeddings — RoPE — work inside attention: every query and key gets rotated by an angle proportional to its position. The further into the sentence a word sits, the more its vectors twist.
Either way, the blindness is cured. Attention can now ask not just which words are relevant, but which came first, which came last, which sat right before this one. Order becomes part of the meaning. Dog bites man is a Tuesday. Man bites dog is a headline. The model finally knows the difference.