Opening the black box
Heads with jobs, single-concept neurons, superposition — the interpretability frontier.
Before this
This opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
We designed every layer and every matrix — the architecture is entirely human-made. But the rules the machine runs on? Nobody wrote those.
Training found these billions of numbers on its own, and they work. They arrive with no comments, no variable names, no documentation — a program no one wrote.
Interpretability researchers treat the weights the way biologists treat a cell: cut in and catalog what's there. And there's structure inside. Attention heads have specific jobs — one exists just to track down what a pronoun refers to.
Individual neurons fire for single concepts — one for the Golden Gate Bridge. And meanings show up as directions in space; that embedding geometry runs through the entire network.
Models represent more ideas than they have dimensions, by overlapping concepts in the same directions. It's called superposition, and it smears the features together — which is what makes them hard to read.
What we can read today is fragments — a head here, a neuron there. But every fragment tells the same story: this isn't a bag of random numbers, it's a program with real structure, one that training wrote and we're slowly learning to read.
We built the machine — every layer, every matrix is human-made — and you now understand all of it. Nobody wrote the rules it runs on, but they aren't magic either: they're weights with meaning, and that meaning can be read. The reading is the work that's left.