Quantizationa giant model on your laptop
Round 16-bit weights into 4-bit buckets: 94 GB becomes 24. Why local LLMs exist.
Next in Frontier AI
DistillationFrontier AI, stop 6 of 102 minThis opens up
Nothing yet. No lesson builds on this one so far.
Transcript
It follows the voice. Click a sentence to jump there.
Every weight in those matrices is stored as a sixteen-bit number. Two bytes each. Cheap, until you count how many weights there are.
Mixtral has forty-seven billion. At two bytes each, that's ninety-four gigabytes. A good consumer GPU holds twenty-four. The model is four times too big for the hardware.
The fix: pick sixteen allowed values, and round every weight to the closest one. Sixteen levels fit in four bits — a quarter of the storage.
Four bits instead of sixteen cuts the model to a quarter of its size — ninety-four gigabytes down to twenty-four. That's the GGUF and llama-dot-cpp world: the same forty-seven billion weights, now running on a MacBook.
The cost is rounding noise — every weight moves a little. Two tricks keep the damage small: each group of weights gets its own scale factor, and the rare extreme values stay at full sixteen-bit precision. Quality barely moves.
Nothing about the model changed — the same forty-seven billion weights, the same matrices, just written in coarser numbers. That one trade takes a model too big for any consumer GPU and fits it on a laptop. Local LLMs exist because of it.