Try first

A gear system. Turning gear A one notch turns gear B three notches. Turning gear B one notch turns gear C two notches.

Turn gear A by one notch. How far does gear C move? Answer, then say how you got it.

Rates multiply along a chain

Six. You multiplied 3 by 2, and that is the entire chain rule.

dC/dA = dC/dB * dB/dA

Read it as: how much C responds to A equals how much C responds to B, times how much B responds to A. Each link contributes its own rate and the rates multiply.

You have already used this twice without ceremony. In lesson 3054 the squaring contributed 2e and the model contributed -x, and you multiplied them. In lesson 3073 the log contributed one rate and the sigmoid another, and they cancelled.

Nothing changes with more links. Four gears means four rates multiplied together.

Applying it to the network

The chain for W2[0][0], the weight from hidden unit 0 to the output.

dL/dW2[0][0] = dL/da2 * da2/dz2 * dz2/dW2[0][0]

Three links, each of which you can work out separately.

dL/da2: how the loss responds to the output. For log loss that is a known formula.

da2/dz2: how the output responds to its pre-activation. That is the sigmoid slope, a2 * (1 - a2), from lesson 3072.

dz2/dW2[0][0]: how the pre-activation responds to the weight. Since z2 = W2[0][0]*a1[0] + W2[0][1]*a1[1] + b2, the answer is a1[0].

So the gradient is the first two multiplied together, times the activation of the unit the weight came from. And for log loss with a sigmoid, the first two collapse to a2 - y by the cancellation in lesson 3073.

dL/dW2[0][0] = (a2 - y) * a1[0]

Error at the output, times the input that weight carried. The same “error times input” shape as every gradient in this course.

Going one layer deeper

For a first layer weight the chain is longer.

dL/dW1[0][0] = dL/dz2 * dz2/da1[0] * da1[0]/dz1[0] * dz1[0]/dW1[0][0]
             = (a2 - y)  *  W2[0][0]  *  a1[0]*(1-a1[0])  *  x[0]

Four links. Read them right to left, which is the direction the computation runs.

x[0] is what this weight was multiplying. a1[0]*(1-a1[0]) is the hidden sigmoid’s slope. W2[0][0] is how strongly the output listens to this hidden unit. (a2 - y) is the error at the end.

Look at the third factor. A hidden unit whose output weight is small barely affects the output, so it receives a small gradient and barely learns. The error is shared out in proportion to how much each unit contributed, which is exactly what credit assignment should mean.

Naming the shared quantity

The first two factors, (a2 - y) * W2[0][0], do not mention W1[0][0] at all. They are the same for every weight coming into hidden unit 0.

Give that a name. Write delta for “how much the loss changes when this unit’s pre-activation changes”.

delta2    = a2 - y
delta1[i] = delta2 * W2[0][i] * a1[i] * (1 - a1[i])

dL/dW2[0][i] = delta2 * a1[i]
dL/dW1[i][j] = delta1[i] * x[j]
dL/db2       = delta2
dL/db1[i]    = delta1[i]

Every gradient is a delta times an input. Compute the deltas once, from the output backwards, and every weight gradient is one multiply away.

That is backpropagation. The next lesson does it with actual numbers.

What to watch

Notice what happens to delta1 when the hidden unit is saturated. If a1[i] is 0.99, then a1*(1-a1) is 0.0099, and the delta is a hundredth of what arrived. Stack three such layers and the gradient at the front is a millionth of the gradient at the back. That is the vanishing gradient problem, and now you can see it as a product of small numbers.

Exercises

  1. Three gears with ratios 2, 0.5 and 4. What is the overall rate? What if one ratio is 0.01?
  2. Write the full chain for b1[1]. How many factors?
  3. If W2[0][0] is exactly 0, what is dL/dW1[0][j]? What does that mean for that hidden unit?
  4. Extend the delta formulas to a network with two hidden layers. You need one more line.
PreviousNext