Try first

You need a function from any number to a number between 0 and 1. Try to build one from parts you know.

Hint: Math.exp(z) is always positive, and it goes from near 0 to enormous as z goes from very negative to very positive. Can you make a fraction out of it that is always less than 1?

Deriving it

Take e = Math.exp(z), which is positive and unbounded above. A fraction of the form e / (e + something positive) is always between 0 and 1, because the top is smaller than the bottom and both are positive.

The simplest choice for the something is 1.

p = exp(z) / (exp(z) + 1)

Check the ends. When z is large, exp(z) is huge and the fraction is just under 1. When z is very negative, exp(z) is nearly 0 and the fraction is just over 0. When z is 0, exp(0) is 1, giving 1 / 2.

All five requirements met. Divide top and bottom by exp(z) and you get the form everyone writes.

function sigmoid(z) {
  return 1 / (1 + Math.exp(-z));
}

This is the sigmoid, also called the logistic function. It was not handed down. It is the simplest thing that meets the list you wrote in the last lesson.

Draw it

view.xMin = -8; view.xMax = 8;
view.yMin = -0.1; view.yMax = 1.1;
clear(); grid(1);
line(-8, 0, 8, 0, "#ccc");
line(-8, 1, 8, 1, "#ccc");
for (let z = -8; z <= 8; z += 0.05) dot(z, sigmoid(z), "#38a", 2);
dot(0, 0.5, "#c33", 5);

The S. Steepest in the middle, flattening towards both ends without ever touching them.

The middle is where the model is unsure, and it is also where the curve is most sensitive: a small change to the score moves the probability a lot. Out at the ends the model is already confident and a large change to the score barely moves the answer. That is sensible behavior and nobody designed it in.

Its derivative is unusually pleasant

We will need the slope of the sigmoid for the chain rule. Work it out and something convenient happens.

sigmoid'(z) = sigmoid(z) * (1 - sigmoid(z))

The slope is expressible using only the output. So if you already computed p = sigmoid(z), the slope costs one multiply and no exponentials. Check it numerically:

const z = 1.3;
const p = sigmoid(z);
log("formula ", (p * (1 - p)).toFixed(8));
log("measured", slopeAt(sigmoid, z).toFixed(8));

Look at what the formula says. The slope is largest at p = 0.5, where it is 0.25, and it falls to nearly zero as p approaches either end.

Read that again, because it is a warning. A confident neuron has almost no gradient. In a deep network, several sigmoids in a row multiply their slopes together, and 0.25 times 0.25 times 0.25 gets small very quickly. That is the vanishing gradient problem from lesson 3060, and this formula is where it comes from. Lesson 3101 is the fix.

What to watch

Math.exp(-z) overflows for very negative z. At z = -800 you get Infinity, and 1 / (1 + Infinity) is 0, which is the right answer by luck. In the other direction it is fine. Libraries use a branching version for safety, and it is worth knowing why they bother.

Exercises

  1. Confirm that sigmoid(-z) equals 1 - sigmoid(z), and say what that symmetry means for the two classes.
  2. Compare the formula and the measured slope at z of 0, 3, and 10. Where do they stop agreeing, and why?
  3. Plot Math.tanh on the same axes. How does it differ from the sigmoid, and what would you have to change to make them match?
  4. At what z does the sigmoid first return exactly 1 in JavaScript? What does that tell you about how confident the model can ever be?
PreviousNext