How language models work

What a language model actually does

One job: guess the next word. Probabilities over every word, sample, append, repeat.

Tokens and embeddings

Text becomes tokens, tokens become vectors. Similar words cluster; directions carry meaning.

The problem attention solves

“River bank” and “bank deposit” start from the identical vector. Meaning must come from neighbors.

The only math you need

Dot product = similarity meter. Matrix × vector = a stack of learned questions. The model IS the matrices.

Neural networks

multiply, add, bend

A neuron multiplies, adds and bends. Stack layers of them into a neural network: one function, set by its weights and biases.

Attention: words talking to each other

Queries, Keys, Values. Scores → softmax → weighted sum. Watch softmax(QKᵀ/√d)·V happen.

Multi-head attention + the MLP

Many heads track many relationships; each token thinks alone in the MLP, where facts live.

The whole Transformer, assembled

Embed → (attend + think) × 96 → unembed → probabilities. The box, opened; the loop, closed.

Mixture of Experts (MoE)

Replace the one big MLP with many experts plus a router picking the top 2. 47B stored, ~13B paid.

Tokenization and BPE

Byte-Pair Encoding merges its way to a vocabulary — and why models can't count the r's in strawberry.

Supervised learning

examples with answers

A line learns house prices from labeled examples, then faces houses it never saw. Text, it turns out, labels itself.

How the matrices get filled in

Weights start as noise. Guess, measure surprise, nudge downhill — trillions of times.

How the model knows word order

Attention is order-blind. Position fingerprints and RoPE put the sequence back.

Also rests on Sine and cosine Calculus

No peeking: causal masking

Mask the upper triangle so the future never leaks — and one sentence trains every position at once.

The KV cache

what context costs

Keys and values get cached; every new token reads a longer shelf. The n² cost, and why long chats slow down.

Pretraining

the internet as flashcards

Every prefix is a question, the next word is the answer. The result: an autocomplete, not an assistant.

From autocomplete to assistant (RLHF)

Fine-tuning teaches the format; human preferences train a reward model that shapes behavior.

Temperature

how models choose

Greedy loops and drones. Temperature sharpens or flattens; top-p cuts the weird tail.

Scaling laws

why bigger got better

Loss vs compute is a straight line on log scales; abilities switch on in jumps.

Opening the black box

Heads with jobs, single-concept neurons, superposition — the interpretability frontier.