Pretrainingthe internet as flashcards
Every prefix is a question, the next word is the answer. The result: an autocomplete, not an assistant.
Before this
Next in How language models work
From autocomplete to assistant (RLHF)How language models work, stop 17 of 202 minThis opens up
Transcript
It follows the voice. Click a sentence to jump there.
One ordinary sentence — the cat sat on the mat — is four quizzes in disguise. The, blank? Cat. The cat, blank? Sat. Every prefix is a question, and the next word is the answer key. That means every sentence ever written is free training data. Nobody had to label a thing.
Pretraining is the same loop from before — guess, measure, nudge — at industrial scale. Fifteen trillion tokens. Months of compute across tens of thousands of GPUs. One objective: predict the next token.
Predicting text well has no cheap shortcut. To get good at it, you're forced to absorb grammar, facts, style, even some reasoning — all smuggled in through the next word.
The result is an autocomplete, not an assistant. Ask the base model for an essay about trees and it keeps the list going: an essay about rivers, an essay about the water cycle. It read a lot of homework assignments, so it wrote one. It continues text. It doesn't answer you.
Pretraining turns every sentence ever written into flashcards. One question — what comes next — gets drilled across trillions of tokens until the model absorbs grammar, facts, and style just to keep guessing well. What you get back isn't an assistant. It's a raw autocomplete that has quietly read the internet.