Try first

You have the sigmoid, log loss, and the fact that dL/dz = p - y. Write the training loop before reading on. It is the loop from lesson 3067 with two changes.

The model

function predictProb(w, b, x) {
  return sigmoid(dotProduct(w, x) + b);
}

function classify(w, b, x, threshold = 0.5) {
  return predictProb(w, b, x) >= threshold ? 1 : 0;
}

Two stages, as promised in lesson 3071. The linear part combines the features, the sigmoid shapes the result. classify is a separate decision made afterwards, and keeping it separate is what lets you change the threshold later without touching the model.

The loss

function logLossAll(w, b, data) {
  let total = 0;
  for (const r of data) total += logLoss(predictProb(w, b, r.x), r.y);
  return total / data.length;
}

The gradient

This is where the cancellation from the last lesson pays off. We know dL/dz = p - y, and z is the same linear combination as before, so dz/dw[j] = x[j]. Chain them.

dL/dw[j] = (1/N) * sum of (p - y) * x[j]
dL/db    = (1/N) * sum of (p - y)
function gradientLogistic(w, b, data) {
  const dw = w.map(() => 0);
  let db = 0;
  for (const r of data) {
    const e = predictProb(w, b, r.x) - r.y;
    for (let j = 0; j < w.length; j++) dw[j] += e * r.x[j];
    db += e;
  }
  return [dw.map(v => v / data.length), db / data.length];
}

Put this next to gradient from lesson 3063 and look at how similar they are. Same loop, same shape, same “error times feature”. The differences are that e is now p - y rather than y - p, which absorbs the minus sign, and there is no factor of 2, which came from squaring.

Two entirely different models, derived from different losses, produce the same gradient shape. That is not an accident either, and it is a good hint that this pattern is more fundamental than either model.

Training

function trainLogistic(data, lr = 0.5, steps = 20000) {
  let w = data[0].x.map(() => 0), b = 0;
  for (let i = 0; i < steps; i++) {
    const [dw, db] = gradientLogistic(w, b, data);
    for (let j = 0; j < w.length; j++) w[j] -= lr * dw[j];
    b -= lr * db;
  }
  return [w, b];
}

const [w, b] = trainLogistic(points);
log("w", w.map(v => v.toFixed(3)).join(" "), " b", b.toFixed(3));
log("log loss", logLossAll(w, b, points).toFixed(4));

let right = 0;
for (const p of points) if (classify(w, b, p.x) === p.y) right++;
log("accuracy", (100 * right / points.length).toFixed(1) + "%");

The update rule has not changed since lesson 3056. It will not change for the rest of the course.

Note the learning rate is 0.5, much larger than the 0.01 that worked for regression. Log loss gradients are bounded, because p - y can never exceed 1 in size, so there is no risk of the runaway from lesson 3059. That is a further quiet benefit of this loss.

Reading the two numbers

Log loss and accuracy measure different things and you want both.

Accuracy counts how many landed on the right side of the threshold. It ignores confidence completely, so a model that scrapes past at 0.51 scores the same as one that is certain at 0.99.

Log loss counts confidence and ignores the threshold. A model can improve its log loss substantially while its accuracy does not move at all, because it became more sure about points it was already getting right.

Training minimizes log loss because it is smooth. You report accuracy because a person can understand it. Watch them diverge and you learn something about your model.

What to watch

If the two classes are perfectly separable, this model never finishes. It can always lower the loss by making the weights larger, which pushes the probabilities closer to 0 and 1. The weights grow without bound and the loss creeps towards zero forever. Our data overlaps, so it settles. Lesson 3098 fixes the general case.

Exercises

  1. Print the loss every 1000 steps. Does it fall smoothly, and does accuracy track it?
  2. Make the two clouds far apart with no overlap. Train for 200000 steps and watch norm(w) grow.
  3. Train with squared error against the sigmoid output instead. Compare how many steps each needs.
  4. Find the three points the model is least sure about. Where are they on the plot?
PreviousNext