Try first

Here is a version of backward with one deliberate bug.

const delta1 = f.a1.map((a, i) => delta2[0] * net.W2[0][i] * a);

Compare it to the correct line. The sigmoid slope is missing its (1 - a) factor.

Question: if you trained a network with this bug, how would you find out? Think about what you would actually observe.

Gradient bugs do not announce themselves

You would probably not find out. The network still trains. The loss still falls, because the gradient is still roughly in the right direction. It converges more slowly and to a slightly worse answer, and both of those look exactly like a model that needs more tuning.

That is what makes backpropagation bugs unusually nasty. No exception, no NaN, no obviously wrong output. Just a model that is quietly worse than it should be, and weeks spent adjusting learning rates.

The fix is not to be careful. The fix is to check, mechanically, every time.

Gradient checking

You have two ways to compute a gradient. The fast one you just wrote, and the slow one from lesson 3087 that measures it directly. The slow one is almost certainly right, because it makes no assumptions about the network at all.

So compare them.

function gradCheck(net, data, h = 1e-5) {
  const analytic = backwardAll(net, data);
  const worst = { name: "", rel: 0 };

  const check = (name, get, set, gradValue) => {
    const before = get();
    set(before + h); const up   = netLoss(net, data);
    set(before - h); const down = netLoss(net, data);
    set(before);
    const numeric = (up - down) / (2 * h);
    const rel = Math.abs(numeric - gradValue) /
                (Math.abs(numeric) + Math.abs(gradValue) + 1e-12);
    if (rel > worst.rel) { worst.rel = rel; worst.name = name; }
    return { numeric, analytic: gradValue, rel };
  };

  for (let i = 0; i < net.W1.length; i++)
    for (let j = 0; j < net.W1[i].length; j++)
      log("W1[" + i + "][" + j + "]", JSON.stringify(
        check("W1", () => net.W1[i][j], v => net.W1[i][j] = v, analytic.dW1[i][j])));

  for (let j = 0; j < net.W2[0].length; j++)
    log("W2[0][" + j + "]", JSON.stringify(
      check("W2", () => net.W2[0][j], v => net.W2[0][j] = v, analytic.dW2[0][j])));

  log("worst relative error", worst.rel.toExponential(2), "at", worst.name);
  return worst.rel;
}

For each parameter: nudge it up, measure the loss, nudge it down, measure again, put it back. Compare the measured slope to the one backpropagation produced.

-2 0 2 -2 0 2 worked out by hand measured by probing the formula is right every point on the line -2 0 2 -2 0 2 worked out by hand the formula has a bug two points are not agreement to about five decimal places is what you are looking for
A wrong gradient does not crash. It trains, slowly and to a worse answer, and looks like a bad learning rate. This plot is how you tell the difference in one minute.

Reading the result

Compare relative error, not absolute. A gradient of 1000 that is off by 0.01 is fine. A gradient of 0.0001 that is off by 0.01 is completely wrong. Dividing by the size of the values makes the comparison meaningful at any scale.

Rough thresholds:

below 1e-7    correct
1e-7 to 1e-4  suspicious, look closer
above 1e-4    there is a bug

Run it on the correct backward and you should see errors around 1e-10. Now introduce the bug from the top of this lesson and run it again. The W1 errors jump to around 0.5, while the W2 errors stay tiny.

That pattern is itself a clue: the layer with the error tells you where to look. A bug in the output delta breaks everything, a bug in the hidden delta breaks only the first layer.

Practical rules

Check on a small network. Two inputs, three hidden units, five rows. The check costs two forward passes per parameter, so a real network takes far too long. The arithmetic is identical, so a small network proves the code.

Check with random weights, not zeros. Zeros make many terms vanish and a broken formula can pass by accident. Use small random values, and use several sets.

Turn it off for training. This is a test, not part of the model. Run it when you write or change the gradient code, then remove it.

Be careful with relu. It has a kink at zero, so if a nudge crosses zero the two sides disagree and the check fails on a correct implementation. Use inputs that keep you away from the kink, or check with a smooth activation instead.

What to watch

This is the same discipline as lesson 3054, where you checked the linear regression gradient against a measured one. It mattered there. It matters far more here, because a two parameter formula can be verified by reading it and a network’s cannot.

Every framework has this built in, and it is worth using even when you did not write the gradient yourself, because a custom loss or a custom layer puts you right back in the same position.

Exercises

  1. Run the check on your correct implementation and record the worst relative error.
  2. Introduce three bugs, one at a time: drop the (1 - a), flip a sign in delta2, and use W2[0][0] where you meant W2[0][i]. Note the error each produces.
  3. Set h to 1e-2 and then 1e-12. Explain both failures using lesson 3053.
  4. Extend the check to cover b1 and b2.
PreviousNext