Try first

Your XOR network gets input [1, 0] and outputs 0.31. The answer should be 1. The network has nine numbers in it.

Which of the nine should change, and in which direction? Try to answer for W1[0][0], the weight from the first input to the first hidden unit. Sit with the difficulty for a moment before reading on.

Why this is harder than before

In section 4 the question was easy. The loss was built directly from m and b, so you could look at the formula and read off how each affected it.

Now W1[0][0] does not appear in the loss at all. Follow it forward:

W1[0][0]  changes  z1[0]
z1[0]     changes  a1[0]
a1[0]     changes  z2[0]
z2[0]     changes  the output
output    changes  the loss

Five hops. And the third hop is shared, because a1[0] feeds the output alongside a1[1], so its effect depends on what the other unit is doing and what the output weights are.

This is called the credit assignment problem. An error appeared at the end and you have to work out how much of it belongs to each of the numbers that contributed. It is the difficulty that held the field up for fifteen years after XOR was understood.

The approach that does not scale

You already have a method. Measure it, with slopeAt from lesson 3053.

function numericalGradient(net, data) {
  const h = 1e-5;
  const g = { W1: [], b1: [], W2: [], b2: [] };
  for (let i = 0; i < net.W1.length; i++) {
    g.W1.push(net.W1[i].map((_, j) => {
      const up = net.W1[i][j] + h, down = net.W1[i][j] - h;
      const before = net.W1[i][j];
      net.W1[i][j] = up;   const a = netLoss(net, data);
      net.W1[i][j] = down; const b = netLoss(net, data);
      net.W1[i][j] = before;
      return (a - b) / (2 * h);
    }));
  }
  // and the same for b1, W2, b2
  return g;
}

Nudge one number up, measure the loss. Nudge it down, measure again. The difference over the gap is that number’s slope. Repeat for all nine.

It works and it is correct. Count the cost. Two full passes over the dataset per parameter, so nine parameters is eighteen passes to take one step. A network with a million parameters needs two million passes for one step, and then thousands of steps.

Nobody has ever trained a real model this way. This is the same problem as lesson 3052, where probing left and right was replaced by a derivative, and the same kind of fix is coming.

The shape of the fix

Notice something about the chain above. Every path from a weight to the loss passes through the output. And every weight in the first layer passes through some hidden unit before that.

So the paths overlap heavily. If you work out how much the loss changes when z2 changes, that single number is useful for every weight in the output layer. Work out how much the loss changes when a1[0] changes, and that number is useful for every weight feeding into hidden unit 0.

The insight of backpropagation is to compute those shared quantities once, starting at the output and working backwards, instead of tracing each parameter’s path separately.

Cost: one forward pass and one backward pass per step, for the whole network, regardless of how many parameters it has. That is the difference between impossible and routine.

What to watch

Keep the numerical version. It is far too slow for training and it is exactly right for checking. Lesson 3091 uses it to verify that the fast version is correct, which is the only reliable way to catch a backpropagation bug.

Exercises

  1. Finish numericalGradient for all four parameter groups.
  2. Run it on the hand built XOR network. Are the gradients near zero? Should they be?
  3. Time one call for a network with 2 inputs and 50 hidden units. Then estimate the time for 784 inputs and 128 hidden.
  4. Write down, in words, the chain of effects from b1[1] to the loss.
PreviousNext