Try first

The true label is 1. Two models answer: one says 0.9, the other says 0.1.

Now the same with squared error. Compute (1 - 0.9)^2 and (1 - 0.1)^2. The second is 81 times worse. Does that ratio feel right to you? Hold your answer.

Second question. The true label is 1 and a model says 0.0001. How bad is that, on a scale where 0.1 was 0.81?

Squared error is too forgiving of confident mistakes

Under squared error, the worst a model can ever score on one example is 1, which happens when it says 0.0 and the answer is 1. So a model that is completely, confidently wrong pays a penalty of 1. A model that hedges at 0.5 pays 0.25.

The gap between those is a factor of four. But those two behaviors are not four times apart in quality. Saying “definitely apple” about an orange is a much worse failure than saying “no idea”, and the loss should say so in stronger terms.

There is a second problem, which is worse in practice. Chain the squared error through the sigmoid and the gradient contains the sigmoid’s slope, which we just saw goes to almost zero when the output is confident. So a model that is confidently wrong gets a tiny gradient and barely corrects itself. It is stuck in exactly the state you most want it to escape.

What we want instead

Write the requirements again.

  • Zero penalty when the model says 1 and the answer is 1.
  • Growing penalty as the answer moves away.
  • Unbounded penalty as the model approaches certainty about the wrong answer.

A function that is 0 at 1 and goes to infinity at 0. That is -log(p).

-log(1.0)    = 0
-log(0.9)    = 0.105
-log(0.5)    = 0.693
-log(0.1)    = 2.303
-log(0.0001) = 9.210

Compare the shape to squared error. Both agree that 0.9 is good. They part company at the bottom: squared error tops out at 1, this one keeps climbing without limit. Certainty about a wrong answer costs unboundedly much, which is the right pricing.

Log loss

That covers labels of 1. For a label of 0 we want -log(1 - p) by the same argument. Combine both cases in one line using the fact that y is 0 or 1.

function logLoss(p, y) {
  const eps = 1e-12;
  const q = Math.min(1 - eps, Math.max(eps, p));
  return -(y * Math.log(q) + (1 - y) * Math.log(1 - q));
}

When y is 1 the second term vanishes and you get -log(p). When y is 0 the first vanishes and you get -log(1 - p). The multiplication by y and 1 - y is a switch written as arithmetic, which is a trick worth recognizing because it appears everywhere in this field.

eps keeps p away from exactly 0 and 1, because Math.log(0) is -Infinity and one such row turns your whole loss into Infinity or NaN. This is the “jumps to NaN in a single step” case mentioned in lesson 3059.

Average it over the dataset and you have log loss, also called binary cross entropy. It is the standard loss for classification and this is the entire reason why.

The part that makes it worth it

Chain log loss through the sigmoid and work out the gradient. Almost everything cancels.

dL/dz = p - y

The gradient with respect to the score is just the error. No sigmoid slope, no factor of two, nothing. The awkward p * (1 - p) term from the sigmoid is exactly cancelled by a matching term from the log.

This is not a coincidence and it is not luck. Log loss is the loss that pairs with the sigmoid, and the cancellation is why. The practical effect is that a confidently wrong model gets a gradient of nearly 1 instead of nearly 0, and corrects itself fast.

What to watch

Log loss is in different units from anything you have seen. It is not a percentage and not an error in the units of the data. A log loss of 0.693 is the score for shrugging at everything, since that is -log(0.5). Anything above that is worse than useless.

Exercises

  1. Plot squared error and log loss against p for y = 1, on one canvas. Mark where they cross.
  2. Compute the log loss of a model that always outputs 0.5, on any dataset. Confirm you get 0.693.
  3. Remove eps and feed in p = 0 with y = 1. Watch what it does to an average over 60 rows.
  4. Verify dL/dz = p - y numerically: pick a z, compute the loss as a function of z, and compare slopeAt to sigmoid(z) - y.
PreviousNext