Mixture of Experts (MoE)
Replace the one big MLP with many experts plus a router picking the top 2. 47B stored, ~13B paid.
Next in How language models work
Tokenization and BPEHow language models work, stop 10 of 202 minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
A Transformer block has two halves, and they're not equally expensive. The MLP — short for multi-layer perceptron — holds most of the parameters, which means most of the knowledge, and most of the bill.
Every token pays that bill in full. A 'the' and a rare chemistry term both run through the entire MLP, in every layer, even though most of that knowledge is irrelevant to either one.
Mixture of Experts splits the one giant MLP into eight smaller ones, and adds a tiny router that decides which token goes where.
For each token, the router picks two experts and splits the work, seventy percent to one, thirty to the other. The remaining six don't run, and skipped computation is free.
A different token lands on a different pair. Nobody assigns these roles: during training each expert drifts toward its own specialty, one absorbing code, another grammar.
You get the knowledge of a huge model at the compute cost of a small one. Mixtral stores forty-seven billion parameters but touches only about thirteen billion per token.
It isn't magic. Every expert still sits in memory, and training has to stop the router from collapsing onto a few favorites. But the bill we started with — every token paying for the whole MLP — is gone. Store forty-seven billion, spend thirteen, and GPT-4 is widely rumored to run on exactly this design.