Tokens and embeddings
Text becomes tokens, tokens become vectors. Similar words cluster; directions carry meaning.
Next in How language models work
The problem attention solvesHow language models work, stop 3 of 201 minTranscript
It follows the voice. Click a sentence to jump there.
Take a small sentence: the robot ate blue cake. The text gets chopped into pieces called tokens, and from here on the model never works with anything smaller than a token.
A token is a common chunk of text. Familiar words stay whole. Rarer ones don't: unbelievable gets carved into un, believ, and able, three pieces the model has seen thousands of times.
Each token gets swapped for a long list of numbers, its embedding, pulled from one giant lookup table. In GPT-3 that list is twelve thousand two hundred eighty-eight numbers long, for every token.
The numbers are coordinates, so each token is a point in a space with thousands of dimensions. Words with similar meanings end up near each other. Cat, dog, and kitten form one neighborhood; cake, pie, and cookie another.
Direction carries meaning too. The step from man to woman is the same step as king to queen: gender is a direction in this space, built into the geometry.
So the robot ate blue cake started as text a computer can't reason about, and ends as points in space where distance is similarity and direction is meaning. Turn words into numbers, and meaning becomes geometry you can do math on.