Distillationteaching small models
The teacher's full distribution is richer than the right answer. 10× smaller, nearly as good.
Before this
Next in Frontier AI
Speculative decodingFrontier AI, stop 7 of 102 minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
Training data is brutally literal. The cat sat on the — label says mat, one hundred percent. Floor, couch, moon: zero, zero, zero. Nothing in that signal says floor was a reasonable guess.
A big teacher model holds a richer answer: mat at forty-six percent, floor at twenty-one, couch at nine, moon down at point three. It knows not just the right word but how wrong every wrong one is — dark knowledge.
Distillation trains a small student to reproduce that whole spread of probabilities, not just the correct answer. The student starts out guessing badly, and its probabilities pull into line with the teacher's. Every correctly ranked alternative is a lesson the hard labels can't teach.
The economics are lopsided. The teacher is huge, slow, and expensive to run; the student has roughly ten times fewer parameters. Trained on the teacher's soft labels, it inherits the conclusions without repeating the discovery — and keeps most of the performance.
This is how the model tiers get made. One frontier-scale teacher, distilled into a mini, a flash, a lite — the fast, cheap versions that handle most of the world's traffic.
So when a small model answers what's the capital of France — Paris — that reflex didn't come from all-or-nothing labels. It's the teacher's dark knowledge, compressed. The small, fast model you use every day is a big model's understanding, poured into ten times fewer parameters.