Neural networksmultiply, add, bend
A neuron multiplies, adds and bends. Stack layers of them into a neural network: one function, set by its weights and biases.
Next in How language models work
Attention: words talking to each otherHow language models work, stop 6 of 202 minTranscript
It follows the voice. Click a sentence to jump there.
Feed two, one, and three into a tiny machine. One number comes out. That machine is an artificial neuron. Each input gets a weight — a number for how much it matters, and which way it pushes. Say the weights are zero point five, minus two, and one.
Multiply each input by its weight. Two times zero point five is one; the others give minus two, then three. Added up, the weighted sum is two. Then add a bias of one — a nudge that ignores the inputs — for a total of three.
Last comes a bend, called an activation function. A common choice: negatives become zero, positives pass through. Three stays three, and a positive output means the neuron fires. Raise the second input from one to three, and the total drops to minus one. The bend makes it zero, and the neuron goes silent.
A layer is many neurons reading the same inputs. Each neuron's weights become one row of a grid called a matrix. Multiply it by the list of inputs — a vector — and each row gives one neuron's weighted sum. Add the biases, apply the bend, and out comes a new list: one number per neuron.
Feed that list into a second layer, and its outputs into a third. That stack of layers is a neural network — one big function, numbers in, numbers out.
Without the bend, stacking gains nothing. Double a number and add one, then triple the result. That's just six times the number, plus three — a straight line. It can never follow an arch; it only tilts and slides. Put the bend back, and each neuron can add a corner. With enough corners, the curve traces the arch.
A network's knobs are its weights and biases, and they set what it computes. Turn one weight from two to minus two, and the output falls from four to two. Learning means finding good values for these knobs. A large language model is a neural network too, with billions of weights, most of them in layers like these.