Multi-head attention + the MLP
Many heads track many relationships; each token thinks alone in the MLP, where facts live.
Before this
Next in How language models work
The whole Transformer, assembledHow language models work, stop 8 of 202 minTranscript
It follows the voice. Click a sentence to jump there.
One attention pattern is called a head, and no layer settles for one. Many run in parallel, each specializing in a different relationship: one matches adjectives to their nouns, another traces a pronoun back to the name it stands for, another ties a verb to whoever's doing it. GPT-3 packs ninety-six into every layer.
Communication is half the job. Once a token has gathered what it needs from its neighbors, it stops listening and processes alone. That's the MLP — short for multi-layer perceptron: the vector expands into a much wider hidden layer, then comes back out at the size it went in. Same shape, sharper meaning.
The MLP is where facts get stored. Feed in Michael Jordan plays, and somewhere in that hidden layer a specific neuron fires for basketball — a single unit encoding an idea the sentence never said.
Both operations run as detours off a main road. A token's vector runs through the model on one long stream. Attention pulls it aside so words can compare notes; the MLP pulls it aside so each word can digest what it heard. Neither replaces the vector — each computes an adjustment and adds it back, and the original signal keeps flowing the whole way through.
Put the pieces together and one block sits on the stream. Many heads let the words talk, each tracing a different relationship; then every word gets its moment to think alone, in the MLP. Communication, then computation — that pair is a Transformer block, and the entire model repeats this same block, over and over.