Frontier AI

Flash attention & the memory wall

The bottleneck is memory traffic, not math. Tile the grid through SRAM — same output, 2-4× faster.

Also rests on Attention: words talking to each other How language models work, The KV cache How language models work

Stretching the context window

Sliding windows, attention sinks, RoPE scaling — how 4k contexts became millions.

Also rests on The KV cache How language models work, How the model knows word order How language models work

Transformers that see

Slice an image into patches, embed each one, run the same blocks. Tokens were never really words.

Also rests on Tokens and embeddings How language models work, The whole Transformer, assembled How language models work

LoRA

fine-tuning at home

Freeze the weights, learn a low-rank sliver ΔW = A·B — 0.1% of parameters, swappable like cartridges.

Also rests on How the matrices get filled in How language models work, The only math you need How language models work

Quantization

a giant model on your laptop

Round 16-bit weights into 4-bit buckets: 94 GB becomes 24. Why local LLMs exist.

Also rests on The only math you need How language models work

Distillation

teaching small models

The teacher's full distribution is richer than the right answer. 10× smaller, nearly as good.

Also rests on How the matrices get filled in How language models work, Temperature How language models work

Speculative decoding

a free speedup

A tiny drafter guesses five tokens; the big model verifies them in one pass. Provably identical output.

Also rests on No peeking: causal masking How language models work, Temperature How language models work

Retrieval

the open-book exam (RAG)

Embed the question, nearest-neighbor a document space, paste what you find into context.

Also rests on Tokens and embeddings How language models work, The whole Transformer, assembled How language models work

Agents

when words become actions

Some words are commands. Think → act → observe, with the context window as working memory.

Also rests on What a language model actually does How language models work, The whole Transformer, assembled How language models work

Diffusion vs autoregression

Predict the next piece, or sculpt the whole canvas from noise? A Transformer stars in both.

Also rests on The whole Transformer, assembled How language models work