Stretching the context window
Sliding windows, attention sinks, RoPE scaling — how 4k contexts became millions.
Before this
Next in Frontier AI
Transformers that seeFrontier AI, stop 3 of 102 minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
Attention runs two tabs at once. The KV cache holds a key and a value for every token it has ever processed, and never lets one go. And each new token's attention work grows with everything before it — total cost, n squared. At those rates a million tokens is wildly expensive. Modern models manage it anyway.
Full causal attention is the culprit. Every token reads its entire past: token eight does eight reads, token eight thousand does eight thousand. Sum that staircase — one plus two up to n — and n squared falls out. Break the staircase and the quadratic breaks with it.
First fix: sliding-window attention. Each token attends only to its last w neighbors — three, here — and anything older drops out of reach. Reads per token stop growing with length, so total work becomes n times w. Linear. Mistral ships this in production.
One catch: the window eventually slides past the beginning of the text, and models get unstable when those first tokens vanish. The fix: let every token keep attending to them. Attention sinks: a window, plus a few tokens nobody is allowed to forget.
Positions hit their own wall. RoPE — short for rotary position embedding — fingerprints each position with waves, and those waves only run out to the longest position seen in training — past that, every coordinate is foreign territory. The workaround is to stretch the waves: same shape, lower frequency, so positions far beyond the old limit still land inside a pattern the model already knows.
A million-token context isn't one invention — it's a stack. Sparse attention, position stretching, and faster kernels, each cutting a different term of the bill; some models drop attention entirely and go state-space. That wildly expensive million tokens from the opening is exactly what this stack tames. The quadratic was never a law of nature. It was just the first design anyone tried.