Diffusion vs autoregression
Predict the next piece, or sculpt the whole canvas from noise? A Transformer stars in both.
Before this
End of this line
Pick a new destinationThis is the last stop in Frontier AI minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
Autoregression is the strategy you already know: write left to right, one token at a time. The, cat, sat, on, the — every word already committed gets a vote on what comes next. All that context converges on a single guess: mat.
Diffusion writes nothing in order. It starts with the full canvas already there, filled with random static. No sky, no ground, no picture. Just noise, everywhere at once.
Then it denoises, one small step at a time. Every pixel updates at once, so structure condenses across the whole image instead of left to right. Sky separates from ground, and a sun forms out of the static.
Images took to diffusion because they're global — every region constrains every other. Language stayed autoregressive, because sentences unfold in order. But the split isn't clean: researchers have built diffusion models that write text, and autoregressive models that paint images.
Inside a modern diffusion model, the denoiser is a Transformer. Chop the noisy image into a grid of patches, treat each patch as a token, and let patches attend to patches — the same mechanism, reused. The arrangement has a name: DiT, the Diffusion Transformer.
So the two philosophies we started with — write left to right, or sculpt the whole thing out of noise — disagree about everything except the engine. One predicts the next piece; the other denoises a whole canvas. Underneath both sits the same machine: the Transformer. Whichever way a model generates, the machine doing the work is one you now understand.