Try first

The smallest possible case. Two neurons in a chain, one input, one output.

x = 1.0
w1 = 0.5, b1 = 0.0        a1 = sigmoid(w1*x + b1)
w2 = 2.0, b2 = -1.0       a2 = sigmoid(w2*a1 + b2)
y = 1.0

Work out the forward pass with a calculator. Then work out dL/dw1 using log loss. Do it before reading on, and keep your numbers.

Forward

z1 = 0.5 * 1.0 + 0.0 = 0.5
a1 = sigmoid(0.5)    = 0.6225

z2 = 2.0 * 0.6225 + (-1.0) = 0.2450
a2 = sigmoid(0.2450)       = 0.5609

loss = -log(0.5609) = 0.5782

The network says 0.56 and the answer is 1, so it is not far off but it should be more confident.

Backward, one step at a time

Start at the output. With log loss and a sigmoid, lesson 3073 gives the shortcut.

delta2 = a2 - y = 0.5609 - 1.0 = -0.4391

Negative, which means increasing z2 would lower the loss. That is right: a larger z2 means a larger a2, closer to the target of 1.

Gradients for the second layer. Each is the delta times what that parameter multiplied.

dL/dw2 = delta2 * a1 = -0.4391 * 0.6225 = -0.2733
dL/db2 = delta2                          = -0.4391

Push the delta back through the weight. How much does the loss change when a1 changes? Through w2.

dL/da1 = delta2 * w2 = -0.4391 * 2.0 = -0.8782

Through the activation. The sigmoid slope at that point is a1 * (1 - a1).

a1 * (1 - a1) = 0.6225 * 0.3775 = 0.2350
delta1 = -0.8782 * 0.2350 = -0.2064

Gradients for the first layer. Same rule as before.

dL/dw1 = delta1 * x = -0.2064 * 1.0 = -0.2064
dL/db1 = delta1                      = -0.2064
x 1 z₁ → a₁ 0.900 → 0.711 z₂ → a₂ -0.453 → 0.389 loss 0.1869 forward backward dL/dw₁ +0.03582 dL/db₁ +0.03582 dL/dw₂ -0.10327 dL/db₂ -0.14526 the two on the left are smaller, because they picked up an extra sigmoid slope on the way
Worth doing once by hand. Forward fills in the top row left to right, backward fills in the bottom row right to left, and nothing else ever happens in a training step.

Check it

function tiny(w1, b1, w2, b2, x = 1, y = 1) {
  const a1 = sigmoid(w1 * x + b1);
  const a2 = sigmoid(w2 * a1 + b2);
  return -Math.log(a2);
}

log("measured dL/dw1", slopeAt(v => tiny(v, 0, 2, -1), 0.5).toFixed(4));
log("measured dL/dw2", slopeAt(v => tiny(0.5, 0, v, -1), 2.0).toFixed(4));
log("measured dL/db1", slopeAt(v => tiny(0.5, v, 2, -1), 0.0).toFixed(4));

You should see -0.2064, -0.2733 and -0.2064. Your hand arithmetic and the measurement agree.

Take one step

lr = 1.0
w1 = 0.5 - 1.0 * (-0.2064) = 0.7064
b1 = 0.0 - 1.0 * (-0.2064) = 0.2064
w2 = 2.0 - 1.0 * (-0.2733) = 2.2733
b2 = -1.0 - 1.0 * (-0.4391) = -0.5609

Recompute the loss with these and you get about 0.42, down from 0.58. Every parameter moved in the direction that helped, including the first layer ones, which are two hops from the loss and never appear in it directly.

What to notice about the sizes

The output layer gradient is -0.2733. The first layer gradient is -0.2064. Smaller, and the shrinkage came from one factor: the sigmoid slope of 0.235.

The largest that factor can ever be is 0.25, at a = 0.5. So every sigmoid layer you pass through multiplies the gradient by at most a quarter, and usually much less. Five layers gives at most 0.25^5, which is about one in a thousand. Ten layers, one in a million.

That is the vanishing gradient problem, in arithmetic you have just done yourself. The front of a deep sigmoid network barely trains. Lesson 3101 is the fix and it is a change to this one factor.

Exercises

  1. Redo the whole thing with y = 0. Which gradients change sign?
  2. Set w2 = 0 and recompute. What is dL/dw1, and why does that make sense?
  3. Add a third neuron in the chain and work out dL/dw1. How much smaller did it get?
  4. Take five steps by hand at lr = 1.0 and watch the loss. Does it fall every time?
PreviousNext