Try first

A network with five hidden layers of sigmoid units. Predict the size of the gradient at the first layer compared to the last, before you measure it.

You worked out the per-layer factor in lesson 3089. Use it.

Build a deep stack and measure

function makeDeep(d, h, layers) {
  const rnd = () => (Math.random() - 0.5) * 2;
  const L = [];
  for (let k = 0; k < layers; k++) {
    L.push({
      W: Array.from({ length: h }, () => Array.from({ length: k ? h : d }, rnd)),
      b: Array.from({ length: h }, () => 0),
    });
  }
  L.push({ W: [Array.from({ length: h }, rnd)], b: [0] });
  return L;
}

function forwardDeepCache(L, x) {
  const acts = [x];
  let a = x;
  for (const layer of L) {
    const z = layer.W.map((row, i) => dotProduct(row, a) + layer.b[i]);
    a = z.map(sigmoid);
    acts.push(a);
  }
  return acts;
}

function backwardDeep(L, x, y) {
  const acts = forwardDeepCache(L, x);
  const grads = [];
  let delta = [acts[acts.length - 1][0] - y];

  for (let k = L.length - 1; k >= 0; k--) {
    const aIn = acts[k];
    grads[k] = delta.map(d => aIn.map(v => d * v));
    if (k > 0) {
      const next = [];
      for (let i = 0; i < aIn.length; i++) {
        let s = 0;
        for (let j = 0; j < delta.length; j++) s += delta[j] * L[k].W[j][i];
        next.push(s * aIn[i] * (1 - aIn[i]));
      }
      delta = next;
    }
  }
  return grads;
}

The loop runs backwards through the layers. At each one it produces the weight gradients, then pushes the delta one layer further back by multiplying through the weights and the sigmoid slope. That is lesson 3088’s formula in a loop.

Look at the sizes

const deep = makeDeep(2, 4, 5);
const grads = backwardDeep(deep, [0.5, -0.3], 1);

grads.forEach((g, k) => {
  const flat = g.flat();
  const mag = Math.sqrt(flat.reduce((s, v) => s + v * v, 0));
  log("layer", k, " gradient size", mag.toExponential(3));
});

Read the numbers from the bottom up. The last layer has a gradient of a reasonable size. Each layer further back is smaller, often by a factor of five or ten, and by the first layer you are somewhere around 1e-4 or below.

Your prediction should have been close. Each layer multiplies by a * (1 - a), at most 0.25 and usually less, and by the weights, which start below 1. Five of those multiplied together is a small number.

Draw it

function showGradients(L, x, y) {
  const grads = backwardDeep(L, x, y);
  const mags = grads.map(g => {
    const f = g.flat();
    return Math.log10(Math.sqrt(f.reduce((s, v) => s + v * v, 0)) + 1e-15);
  });
  view.xMin = -0.5; view.xMax = L.length - 0.5;
  view.yMin = Math.min(...mags) - 1; view.yMax = 1;
  clear(); grid(1);
  for (let k = 0; k < mags.length; k++) {
    line(k, view.yMin, k, mags[k], "#38a");
    dot(k, mags[k], "#c33", 4);
  }
}

showGradients(deep, [0.5, -0.3], 1);

Plotted on a log scale, because a linear scale would show one visible bar and four invisible ones. The bars fall off in a straight line, which on a log scale means the gradient shrinks by a constant factor per layer. Exponential decay, layer by layer.

What this means for training

The first layer’s weights change ten thousand times more slowly than the last layer’s. They are effectively frozen at their random starting values while the layers above them learn.

That is bad in a specific way. The early layers are supposed to learn the most general features, the ones everything else builds on. If they never move, the network is a deep stack of random projections with one trained layer on top.

The reverse also happens. Start with large weights and the same product runs the other way, growing by a factor per layer until the gradient is enormous and the first update destroys the model. That is exploding gradients, and it appears as a loss that becomes NaN within a few steps.

Both come from the same source: a product of many factors. Products of many numbers below one collapse, products of many numbers above one blow up, and hitting the narrow band in between is not something you get by luck.

What to watch

Print gradient sizes per layer whenever a deep network trains badly. It separates three failures that otherwise look identical: vanishing at the front, exploding anywhere, and a learning rate that is simply wrong. Lessons 3100 and 3101 are the two standard fixes, and both are aimed directly at this plot.

Exercises

  1. Try 2, 5, 10 and 20 layers. Plot the first layer gradient against depth.
  2. Scale the initial weights by 4. Does the plot invert?
  3. Replace the sigmoid with tanh, whose slope reaches 1 rather than 0.25. How much does it help?
  4. Replace it with relu, whose slope is exactly 1 where it is positive. Compare all three.
PreviousNext