Try first

You have the delta formulas from lesson 3088 and you have done one by hand in lesson 3089. Write backward(net, x, y) before reading on.

The code

function backward(net, x, y) {
  const f = forward(net, x);

  // output layer
  const delta2 = [f.a2[0] - y];
  const dW2 = [f.a1.map(a => delta2[0] * a)];
  const db2 = [delta2[0]];

  // hidden layer
  const delta1 = f.a1.map((a, i) => delta2[0] * net.W2[0][i] * a * (1 - a));
  const dW1 = delta1.map(d => x.map(xj => d * xj));
  const db1 = delta1.slice();

  return { dW1, db1, dW2, db2 };
}

Nine lines, and every one of them is a formula from lesson 3088 written out.

delta2 is the output error, using the log loss shortcut. dW2 is that delta times each hidden activation. delta1 pushes the error back through W2 and then through the sigmoid slope. dW1 is each hidden delta times each input.

The pattern to hold on to: a weight’s gradient is always the delta at its destination times the activation at its source. That is true in every layer of every network of this kind.

Over the whole dataset

function backwardAll(net, data) {
  const acc = {
    dW1: net.W1.map(r => r.map(() => 0)),
    db1: net.b1.map(() => 0),
    dW2: net.W2.map(r => r.map(() => 0)),
    db2: net.b2.map(() => 0),
  };
  for (const r of data) {
    const g = backward(net, r.x, r.y);
    for (let i = 0; i < acc.dW1.length; i++) {
      for (let j = 0; j < acc.dW1[i].length; j++) acc.dW1[i][j] += g.dW1[i][j];
      acc.db1[i] += g.db1[i];
    }
    for (let j = 0; j < acc.dW2[0].length; j++) acc.dW2[0][j] += g.dW2[0][j];
    acc.db2[0] += g.db2[0];
  }
  const k = 1 / data.length;
  acc.dW1 = acc.dW1.map(r => r.map(v => v * k));
  acc.db1 = acc.db1.map(v => v * k);
  acc.dW2 = acc.dW2.map(r => r.map(v => v * k));
  acc.db2 = acc.db2.map(v => v * k);
  return acc;
}

Run the backward pass for every row, add the gradients up, divide by the count. Averaging matches the loss, which is also an average, so the gradient corresponds to the thing being minimized.

The update

function step(net, g, lr) {
  for (let i = 0; i < net.W1.length; i++) {
    for (let j = 0; j < net.W1[i].length; j++) net.W1[i][j] -= lr * g.dW1[i][j];
    net.b1[i] -= lr * g.db1[i];
  }
  for (let j = 0; j < net.W2[0].length; j++) net.W2[0][j] -= lr * g.dW2[0][j];
  net.b2[0] -= lr * g.db2[0];
}

function trainNet(net, data, lr = 1.0, steps = 20000) {
  for (let i = 0; i < steps; i++) step(net, backwardAll(net, data), lr);
  return net;
}

The update rule from lesson 3056, unchanged. It has not changed since the two parameter version, and it will not change again. Every difference between a two parameter model and a large network is in how the gradient is computed, not in what is done with it.

The cost

One forward pass and one backward pass per row, per step. The backward pass does roughly the same amount of arithmetic as the forward pass, so training costs about three times what predicting costs.

Compare that to the numerical version in lesson 3087, which needed two full passes per parameter. For nine parameters, backpropagation is about six times faster. For a million parameters it is about two hundred thousand times faster, and the ratio keeps growing.

That difference is why this algorithm mattered enough to restart a field.

What to watch

The output delta a2 - y is a shortcut that assumes log loss with a sigmoid output. Change either one and it is wrong. With squared error the delta is 2*(a2 - y) * a2 * (1 - a2), and you would notice the difference as a model that trains much more slowly rather than as an error.

Never trust this code until you have checked it. The next lesson does exactly that.

Exercises

  1. Rewrite backward to take the activation function and its derivative as arguments, so you can swap in relu.
  2. Add a second hidden layer. Which lines do you copy, and what changes in them?
  3. Count the multiplications in one backward call for 784 inputs and 128 hidden units.
  4. What happens if you forget the 1 / data.length? Try it and see which hyperparameter you would have to change to compensate.
PreviousNext