Try first

The sigmoid’s slope is a * (1 - a), at most 0.25. Every layer multiplies the gradient by it, so ten layers multiply by at most 0.25^10.

Work that number out. Then say what property an activation function would need for the problem to disappear entirely.

The requirement

0.25^10 is about one in a million. So the first layer of a ten layer sigmoid network receives a gradient a million times smaller than the last, and that is the best case.

The property you want is a slope of 1. Then the layers multiply by 1 each time and nothing shrinks. Combined with the requirement from lesson 3082 that it be non-linear, that sounds contradictory: a function with slope 1 everywhere is a straight line.

The resolution is that it does not need slope 1 everywhere. It needs slope 1 where it is active, and something else elsewhere.

function relu(z) { return z > 0 ? z : 0; }
function reluSlope(z) { return z > 0 ? 1 : 0; }

Positive input, pass it through with slope exactly 1. Negative input, output 0 with slope 0. Non-linear, because of the kink at zero. And gradients pass through the positive side without shrinking at all.

It is hard to overstate how much this one change mattered. Deep networks were considered impractical for years, and this function, which is Math.max(0, z), is a large part of why they became routine.

Measure it

for (const f of ["sigmoid", "tanh", "relu"]) {
  const net = makeDeepWith(2, 8, 8, f);
  const mags = layerGradientSizes(net, [0.5, -0.3], 1);
  log(f.padEnd(8), mags.map(v => v.toExponential(1)).join("  "));
}

Sigmoid falls off a cliff, dropping by a factor of five or more per layer. tanh is better, because its slope reaches 1 at the middle rather than 0.25, but it still saturates at the ends. relu holds roughly steady across all eight layers.

That flat line is the whole reason for the switch.

The cost: dead units

relu has a failure of its own, and it is worth knowing because it is easy to miss.

If a unit’s pre-activation is negative for every row in your data, it outputs 0 every time. Its slope is 0 every time, so its gradient is 0, so its weights never change, so it stays negative forever. The unit is dead and will not recover.

A large learning rate can kill units in batches by pushing a whole layer’s biases sharply negative in one step. Losing a few percent is normal. Losing forty percent means the network has quietly lost most of its capacity.

function deadFraction(net, data) {
  const counts = net.W1.map(() => 0);
  for (const r of data) {
    const f = forward(net, r.x);
    f.a1.forEach((a, i) => { if (a > 0) counts[i]++; });
  }
  return counts.filter(c => c === 0).length / counts.length;
}

Check this after training. It is two lines and it explains a lot of otherwise mysterious underperformance.

The usual fix

function leakyRelu(z, alpha = 0.01) { return z > 0 ? z : alpha * z; }
function leakySlope(z, alpha = 0.01) { return z > 0 ? 1 : alpha; }

Give the negative side a small slope instead of zero. A unit that goes negative still receives a small gradient, so it can climb back out. The cost is one extra multiply and one hyperparameter that almost nobody tunes.

There are smoother variants, ELU and GELU among them, which trade a little arithmetic for a continuous derivative. GELU is what most current large language models use. The differences between them are real and small compared to the difference between any of them and the sigmoid.

Where the sigmoid still belongs

It is not obsolete, it is just no longer a hidden layer activation.

Use it on the output of a binary classifier, where you want a probability and lesson 3073 showed how neatly it pairs with log loss. Use softmax, its multi-class relative, on the output of a multi-class classifier, which is section 11. Use it inside gates in recurrent architectures, where a value between 0 and 1 is meant to act as a proportion.

The rule: sigmoid where you need a bounded output with a meaning. relu or a variant everywhere in between.

What to watch

Switching to relu means switching to He initialization from lesson 3100. The two go together, because He’s factor of two exists precisely because relu discards half the signal. Use Xavier with relu and the activations shrink layer by layer, which undoes much of what you switched for.

Exercises

  1. Modify backward to take an activation and its derivative as arguments. Retrain XOR with all three.
  2. Train a 6 layer network with sigmoid and with relu. Plot both loss curves on one canvas.
  3. Train with relu at a deliberately high learning rate and report deadFraction. Then switch to leaky and repeat.
  4. Plot relu, leaky relu and their slopes on one canvas. Mark the point where the slope is undefined.
PreviousNext