Speculative decodinga free speedup

0:03The sequential bottleneck

Generation has a built-in bottleneck: every new token costs one full forward pass of the big model. Billions of parameters fire, and out comes one word. Then the whole machine runs again for the next word, and again for the one after that.

0:18A tiny draft model

The workaround starts with a second network — a tiny draft model, fast and a little sloppy. In the time the big model takes to produce a single token, the draft fires off five guesses.

0:31Checking guesses in one pass

Those five guesses go to the big model together, and it checks all of them in one forward pass. Causal masking means it was scoring every position anyway, so verifying a whole batch of tokens costs about the same as generating one.

0:45Keep the prefix, fix the miss

Wherever a guess matches what the big model would have said, it stays. At the first disagreement, the draft's token is thrown out and the big model's own pick goes in. That one expensive pass just bought four real tokens.

1:00The loop and the worst case

The loop starts over, with the draft guessing again from the new prefix. Worst case — every guess wrong — a pass still gets you one genuine token, so you never fall behind plain decoding.

1:12Seven passes versus two

For seven tokens, the old way charges seven full passes of the big model. Speculative decoding delivers the same seven for two big passes, plus a handful of draft steps cheap enough to round to zero.

1:25Faster, provably identical output

So the bottleneck breaks. Two to three times the speed, and the odds for every word come out provably identical — not similar, identical. Same tokens the big model would have written alone, just verified in parallel instead of waited on one at a time. The only real cost is keeping a small draft model running next to the big one.