The KV cachewhat context costs
Keys and values get cached; every new token reads a longer shelf. The n² cost, and why long chats slow down.
Before this
Next in How language models work
PretrainingHow language models work, stop 16 of 202 minTranscript
It follows the voice. Click a sentence to jump there.
When a transformer processes a token, it computes that token's key and value once, then stores them. An old token never has to be recomputed.
That pays off when a new token arrives. Instead of reprocessing the whole sentence, its query reads the stored keys and values of everything before it. Compute once, reuse — that's the KV cache, K V for those keys and values.
There's a catch. Each new token reads everything stored so far, and everything-so-far keeps growing. Token one hundred does a hundred reads. Token ten thousand does ten thousand. The cache never shrinks.
The work per token climbs in a straight line — each does a little more reading than the last. Total work is the sum of that line, and summing a line gives a square. That's the n squared cost of attention.
Which is why context windows have hard limits, why long chats turn sluggish, and why a long document costs real money to process. Context isn't storage the model fills. It's work the model repeats for every new token, and that work grows with the square of the length.