Chapter 4 · 6 min read

Neural networks and deep learning

In one sentence: A neural network is many tiny voters joined in layers, and it learns by nudging how much say each voter has, a little at a time, until its answers improve.

Why do we need them?

Some jobs are too tangled for hand-written rules. How would you write rules to tell a cat from a dog in a photo? Or to understand a spoken sentence? There are too many little clues, and they mix in too many ways.

A neural network is a design for a model that can learn such tangled patterns by itself. The name comes from a loose idea of how brain cells connect. But a neural network is not a brain. It is a large pile of simple arithmetic.

One neuron: a tiny voter

Let us meet one neuron. It is a tiny voter that asks one small question.

Say it must decide: is this picture an apple?

One artificial neuron. Three clues, is it round, is it red and does it have a stem, each have a score and a weight. They are added up to 4.5, a nudge of minus 3 gives 1.5, and the decide step says yes, so the output is apple.

1. It receives clues as numbers. "Is it round?" scores 0.9. "Is it red?" scores 0.8. "Has a stem?" scores 0.2. 2. Each clue has a weight, which says how much say that clue has. Roundness counts for 3, redness for 2, and the stem for 1. 3. It adds up each score times its weight: 0.9 × 3 + 0.8 × 2 + 0.2 × 1 = 4.5. 4. It adds a nudge, called the bias. Here the nudge is −3, which makes the neuron a little harder to convince. The result is 1.5. 5. It decides. A rule called the activation says whether the neuron speaks up, and how loudly. Here 1.5 is big enough, so it says "apple".

The weights are the key. A different set of weights makes a different voter. Learning means finding good weights.

Why the "decide" step matters

Without the decide step, a whole network would be only addition and multiplication, and it could only learn straight-line patterns. The decide step bends the line. It lets networks learn curves, shapes, faces and sentences. It is a small piece with a big job.

Layers

One voter is not clever. But connect thousands of them in rows called layers, and something new appears.

The input layer takes in the raw numbers, such as the dots of a photo.
Several hidden layers sit in the middle. Each neuron listens to the layer before it.
The output layer gives the final answer, such as "apple".

In an image network, the first layers tend to notice simple things like edges. Later layers combine edges into shapes, and shapes into objects. Nobody tells the network to do this. It works out useful clues from the examples.

Deep learning just means a neural network with many hidden layers. "Deep" means "many layers".

How the network learns

At the start, the weights are random, so the network guesses badly. Then it uses the same loop from Chapter 3: guess, check, adjust. Let us make each step clearer.

Guess. A picture goes in, layer by layer, and an answer comes out. This is the forward pass.
Check. A score called the loss says how wrong the answer was.
Pass the blame backwards. The network works out how much each weight contributed to the mistake. It starts at the output and goes back towards the input. This is called backpropagation.
Adjust. Each weight is moved a tiny amount in the direction that would have reduced the mistake.

A curve shaped like a valley. A ball starts high on one side with big mistakes and takes smaller and smaller steps down towards the bottom, where the mistakes are smallest.

Picture yourself on a hillside in thick fog. You cannot see the valley. But you can feel which way is downhill under your feet. So you take a small step downhill, then feel again, and step again. That is exactly what the network does with its weights. How big each step is has a name too: the learning rate. Steps that are too big make you overshoot. Steps that are too small take forever.

One full trip through all the study examples is called an epoch. Training usually takes many epochs.

Why now?

Neural networks were invented decades ago. They became useful recently because two things arrived together: huge amounts of data, and very fast chips that can do millions of small sums at once. Graphics chips, first made for video games, turned out to be perfect for this.

Go deeper: Deep dive B: inside a neural network works one tiny network by hand: the forward pass, the loss, backpropagation and the weight update, with every number shown.

Back to Asha

Asha wants her website to show which cakes are on her counter by looking at a photo. A neural network can learn this from several thousand photos of her cakes, each labelled. Nobody writes a rule such as "a round shape with pink on top is a strawberry cake". The network works out its own clues.

Words explained

Neuron: one tiny voter that adds up weighted clues and decides how loudly to speak.
Weight: how much say one clue has. The things a network learns are mostly weights.
Bias: a nudge that makes a neuron more or less easy to convince.
Activation: the decide step that lets networks learn curves, not only straight lines.
Layer: a row of neurons. A network has an input layer, hidden layers and an output layer.
Loss: a number saying how wrong the network's answer was.
Epoch: one full pass through the study examples.

The limits of this chapter

This is a simplified picture of one neuron. Real networks use many more kinds of layers and tricks.
Nobody fully understands why a trained network gives a particular answer. Its knowledge is spread across millions of weights. This is why careful testing matters so much in the main book.
Networks need a lot of examples, and a lot of computing power, to learn well.

For engineers

A neuron computes an activation function applied to a weighted sum plus a bias. Common activations are ReLU, sigmoid, tanh and softmax (the last turns scores into probabilities that add up to 1). Non-linear activations are what give a network more power than a single linear model.
Forward propagation computes outputs layer by layer. A loss function compares the output with the target. Backpropagation applies the chain rule to compute how the loss changes when each weight and bias changes (the gradient). An optimiser such as stochastic gradient descent or Adam moves the parameters against the gradient, scaled by the learning rate. Data is processed in mini-batches, and one pass over the dataset is an epoch.
Recurrent neural networks (RNNs) process sequences step by step. Their gradients can shrink as they are multiplied across many steps (the vanishing gradient problem), so they struggle to learn long-range links. That is one reason attention-based models replaced them for language (Chapter 6).
Parameter counts are reported in millions or billions. The adjustable numbers in a model are called its parameters or weights, and the two words are often used as if they mean the same thing.

Check yourself

What does a weight do? Answer: It sets how much say one clue has in a neuron's decision.
How does a network learn? Answer: It guesses, measures how wrong it was, works out which weights to change, and nudges them a little, many times over.
What does "deep" mean in deep learning? Answer: The network has many layers.

Plain-words explanations in the author's own words. The examples are invented, and no real company, product or customer is described. Simplified on purpose: where a detail matters, the "For engineers" part says so. Disclaimers