Transformers that see
Slice an image into patches, embed each one, run the same blocks. Tokens were never really words.
Before this
Next in Frontier AI
LoRAFrontier AI, stop 4 of 102 minThis opens up
Transcript
It follows the voice. Click a sentence to jump there.
A Transformer eats sequences of tokens — one token after another. A photograph is the opposite: an eight-by-eight grid of pixels has no beginning, no end, no reading order. For this machine to handle images, someone has to figure out how to tokenize a picture.
The answer is simple: cut the image into sixteen patches, and call each patch a token. No new machinery — just a declaration. The photo is now a sixteen-word sentence.
Take one patch — the cat's ear. A single shared matrix flattens it into a vector, and a position fingerprint gets added to it, so the model knows where in the image that patch came from. The result is a patch embedding: the visual cousin of a word embedding.
From there, nothing is new. Attention, then the MLP — short for multi-layer perceptron — stacked N times: the same blocks that handle language. The sixteen patch tokens feed straight in, and not a line of the architecture has to change.
Attention now means patches interrogating patches. The ear asks the tail whether they belong to the same animal — one cat, or two? Same mechanism, same math. Geometry instead of grammar.
So a photograph was never really the opposite of a sentence. Tokens never had to be words. Image patches, audio slices, protein letters, game moves — chop anything into a sequence, and the exact same Transformer predicts what comes next. The Transformer never knew it was reading. It only knew tokens, and never asked what they meant.