Tokenization and BPE
Byte-Pair Encoding merges its way to a vocabulary — and why models can't count the r's in strawberry.
Next in How language models work
Supervised learningHow language models work, stop 11 of 202 minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
One token per word sounds reasonable until you try it. Run, running, runs, ran, runner — each variant needs its own slot, and so does every rare word, every typo, every German compound noun. The vocabulary explodes.
The other extreme, one token per letter, keeps the vocabulary tiny. But strawberry becomes ten tokens, every sentence balloons, and attention pays its quadratic cost on all that extra length.
The compromise is Byte Pair Encoding: start from individual characters, find the neighboring pair that appears most often, and merge it into a new token. Then do it again.
In banana, the pair a-n appears most, so it merges first. Then an plus a becomes ana, b joins on, and six characters collapse into one token. Scale that to millions of merges over real text, and anything frequent — whole words, prefixes, common endings — earns its own slot in the vocabulary.
The vocabulary ends up with a frequency bias. Common words travel as one piece — strawberry made the cut. Rarer words get assembled from parts: unbelievable ships as un, believ, able — three chunks the model glues back together.
Which explains a famous failure. Ask a model how many r's are in strawberry and it flails, because the word arrives as one indivisible token — the letters were never handed over. Byte Pair Encoding sits between one slot per word and one slot per letter, and the seams it draws set hard limits: a model can only reason about the pieces tokenization hands it. It can't count what it was never given.