The only math you need
Dot product = similarity meter. Matrix × vector = a stack of learned questions. The model IS the matrices.
Before this
Nothing. This is a starting point: it needs no other lesson first.
Next in How language models work
Neural networksHow language models work, stop 5 of 202 minTranscript
It follows the voice. Click a sentence to jump there.
A token is a vector — a list of numbers. And any list of numbers is geometry: two numbers make an arrow in a plane, so 'cat' becomes zero point eight, zero point three — a direction you can point to.
Nearly everything a transformer does reduces to one operation: the dot product. Multiply two vectors entry by entry, add it all up, and two vectors collapse into a single number. Cat dot dog gives you one number, and that number means something.
It measures agreement. Vectors pointing the same way score big and positive. A vector for 'car' lands near zero. A true opposite goes negative. The dot product is a similarity meter.
Attention is built on this. A query dotted with a key is one word asking another: how relevant are you to me? The scores that route information through a transformer are multiply-and-add, nothing more.
One step up is a matrix times a vector. A matrix is a grid of numbers the model learned during training, and it acts like a machine — vector in, different vector out.
Each output entry is a single dot product — one row of the matrix against the incoming vector. Three rows, three answers. A matrix is a stack of learned questions, asked at once.
Which reframes the architecture. Embedding words, making queries, keys, and values, the two layers of the MLP — short for multi-layer perceptron — scoring the final output — every stage of the transformer is one of these matrices doing its one multiply.
The model is the matrices. The architecture decides where they sit; training fills in their numbers — a hundred and seventy-five billion of them in GPT-3. Every answer it has ever given you was multiply, add, repeat. That's the only math you need.