Try first

The model outputs 0.5 exactly when the score is 0, which is when w0*x0 + w1*x1 + b = 0.

That is an equation in two unknowns. Rearrange it into y = mx + c form before reading on. What you get tells you the shape of the boundary.

The boundary is a straight line

w0*x0 + w1*x1 + b = 0
x1 = -(w0 * x0 + b) / w1

A straight line, with slope -w0/w1 and intercept -b/w1. So logistic regression draws a line and calls everything on one side class 1.

The sigmoid never changes that. It changes how confident the model is as you move away from the line, but the line itself is where the linear part hits zero, and a linear part gives a straight boundary. That limit is the whole reason section 7 exists.

Draw the confidence, not just the line

function decisionMap(w, b, xMin, xMax, yMin, yMax) {
  const wd = canvas.width, ht = canvas.height;
  const img = ctx.createImageData(wd, ht);
  for (let py = 0; py < ht; py++) {
    for (let px = 0; px < wd; px++) {
      const x0 = xMin + (px / wd) * (xMax - xMin);
      const x1 = yMax - (py / ht) * (yMax - yMin);
      const p = predictProb(w, b, [x0, x1]);
      const i = (py * wd + px) * 4;
      img.data[i]     = Math.round(60 + 195 * p);
      img.data[i + 1] = Math.round(140 + 60 * (1 - Math.abs(p - 0.5) * 2));
      img.data[i + 2] = Math.round(230 - 180 * p);
      img.data[i + 3] = 255;
    }
  }
  ctx.putImageData(img, 0, 0);
  view.xMin = xMin; view.xMax = xMax;
  view.yMin = yMin; view.yMax = yMax;
}

decisionMap(w, b, 0, 10, 0, 10);
for (const p of points) dot(p.x[0], p.x[1], p.y ? "#402" : "#024", 4);

Every pixel is colored by the probability the model assigns to that position. The result is a smooth gradient from one color to the other, with a band of uncertainty running through the middle where the probability is near 0.5.

Draw the data on top and check it. Most orange points should sit in the orange region. The ones in the pale band are the ones the model is unsure about, and some of them will be on the wrong side.

Watch it learn

function trainAnimated(data, lr = 0.5, steps = 3000) {
  let w = data[0].x.map(() => 0), b = 0, i = 0;
  const tick = () => {
    for (let k = 0; k < 20 && i < steps; k++, i++) {
      const [dw, db] = gradientLogistic(w, b, data);
      for (let j = 0; j < w.length; j++) w[j] -= lr * dw[j];
      b -= lr * db;
    }
    decisionMap(w, b, 0, 10, 0, 10);
    for (const p of data) dot(p.x[0], p.x[1], p.y ? "#402" : "#024", 4);
    out.textContent = "step " + i + "  loss " + logLossAll(w, b, data).toFixed(4);
    if (i < steps) requestAnimationFrame(tick);
  };
  tick();
}

trainAnimated(points);

requestAnimationFrame runs 20 training steps, redraws, and hands control back to the browser so the picture actually appears. Doing all 3000 steps in a tight loop would freeze the page and show you only the final frame.

What you should see

Two distinct phases, and the split between them is the point of this lesson.

First the boundary rotates into position, quickly. The weights are finding the right direction, which is the direction that separates the classes.

Then, once the angle is right, the picture keeps changing but the line stops moving. The color bands get narrower, the transition sharpens, and the loss keeps falling. The weights are growing in size while keeping their direction, which is the model becoming more confident about a boundary it already found.

Direction of w sets where the line is. Length of w sets how sharply the model commits. Those are two separate things and the animation separates them for you.

What to watch

The band of uncertainty narrows for as long as you keep training. Left alone it becomes a hard edge, and the model claims near certainty right up to the boundary. That is usually overconfidence rather than knowledge, and it is one of the things regularization in lesson 3098 holds back.

Exercises

  1. Print norm(w) and the angle of w every 200 steps. Confirm the two phases numerically.
  2. Start from large random weights instead of zeros. Does the animation look different?
  3. Draw the 0.5 line explicitly using the rearranged formula, on top of the map. Do they agree?
  4. Move one class far away so the data separates cleanly. Watch the band collapse and the weights run away.
PreviousNext