From autocomplete to assistant (RLHF)
Fine-tuning teaches the format; human preferences train a reward model that shapes behavior.
Before this
Next in How language models work
TemperatureHow language models work, stop 18 of 202 minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
Step one is supervised fine-tuning. People write thousands of example conversations — a human asks, an assistant answers — each one written the way the model should behave. Then the pretrained model trains on those transcripts.
Nothing here is new — same training loop, same loss, just a tiny hand-picked dataset. What the model absorbs is the format: a question arrives, an answer follows, and the answer is supposed to be useful.
Step two: humans grade. Give the model one prompt — explain gravity to a five-year-old — and have it write two answers. One is technically correct spacetime curvature; the other says the Earth gives everything a gentle hug. A person picks. The hug wins. Repeat thousands of times.
Those judgments train a second network — a reward model that predicts what a human would prefer. Then the assistant gets nudged, answer by answer, toward whatever that model scores highly.
That's RLHF — reinforcement learning from human feedback. It takes the model from autocomplete to assistant, and the architecture never changes: the same matrices do the same math. Nothing is rebuilt — only the behavior gets shaped. A raw next-word predictor, taught the format and pointed at what people prefer, becomes the thing answering you now. Exactly how I got here.