Scaling lawswhy bigger got better
Loss vs compute is a straight line on log scales; abilities switch on in jumps.
Before this
Next in How language models work
Opening the black boxHow language models work, stop 20 of 202 minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
Plot a model's prediction error against the compute spent training it, both on log scales, and you get a straight line. Not roughly straight — straight, and it holds across ten orders of magnitude.
GPT-2, GPT-3, GPT-4 — each one lands right where the line said it would. Nothing about the architecture changed between them. It's the same machine underneath, just bigger matrices and more data.
Loss falls smoothly, no cliffs anywhere in the curve. But abilities arrive in jumps. At some scale a model gets arithmetic. Then translation. Then code. Then whole chains of reasoning. Smooth input, lumpy output — and nobody fully understands why.
That straight line says intelligence can be bought in bulk: pour in more compute, and the error keeps falling, exactly on schedule. Somewhere along the way, arithmetic and code and reasoning switch on for free. No new idea required — just a bigger version of the same machine. The world spent billions on that one line, and every time, it held.