No peeking: causal masking
Mask the upper triangle so the future never leaks — and one sentence trains every position at once.
Next in How language models work
The KV cacheHow language models work, stop 15 of 201 minThis opens up
Transcript
It follows the voice. Click a sentence to jump there.
Attention, written out as a table: one row per word, scoring how much it cares about every other word in the sentence. All the weight sits on words that came earlier.
During training, the model's job is to guess each word before it arrives. Predict sat from just The cat. But the full sentence gets processed all at once, and the answer, the very next word, is sitting in plain view. That's copying, not learning.
Every score above the diagonal gets set to minus infinity before the softmax, and softmax turns minus infinity into exactly zero. Each word draws on its own past, nothing else.
Because no row can cheat, every row becomes a fair exam question. One sentence, one pass through the network, and the model gets graded on five predictions at once.
So the rule that started as no peeking is this triangle of zeros. Blank out the future, and every word is forced to earn its prediction from its own past alone. That one constraint is why GPT-style models are called causal, or autoregressive: information flows one way, and the future never leaks into the past.