Try first

This lesson has no new theory. You are going to draw your own datasets and find out where logistic regression stops working.

Before you build it, predict which of these it will handle: two separated blobs, two blobs that overlap, one blob inside a ring, two diagonal stripes. Commit to an answer for each.

Build the tool


  class: A
         B
  
  

let drawn = [];
let model = null;

canvas.onclick = (ev) => {
  const rect = canvas.getBoundingClientRect();
  const px = ev.clientX - rect.left;
  const py = ev.clientY - rect.top;
  const x0 = view.xMin + px / canvas.width  * (view.xMax - view.xMin);
  const x1 = view.yMax - py / canvas.height * (view.yMax - view.yMin);
  const cls = document.querySelector('input[name=c]:checked').value | 0;
  drawn.push({ x: [x0, x1], y: cls });
  redraw();
};

function redraw() {
  if (model) decisionMap(model[0], model[1], 0, 10, 0, 10);
  else { clear(); grid(1); }
  for (const p of drawn) dot(p.x[0], p.x[1], p.y ? "#e80" : "#38a", 4);
  out.textContent = drawn.length + " points";
}

document.getElementById("go").onclick = () => {
  if (drawn.length < 4) return;
  model = trainLogistic(drawn, 0.5, 20000);
  const s = scores(confusion(model[0], model[1], drawn));
  redraw();
  out.textContent += "   precision " + s.precision.toFixed(2) +
                     "   recall " + s.recall.toFixed(2);
};

document.getElementById("clr").onclick = () => { drawn = []; model = null; redraw(); };

view.xMin = 0; view.xMax = 10; view.yMin = 0; view.yMax = 10;
redraw();

The click handler is the only fiddly part. getBoundingClientRect gives where the canvas sits on the page, so subtracting it turns a page position into a canvas pixel. Then the same conversion as toScreen from lesson 3034, run backwards, turns a pixel into data coordinates.

Four experiments

Do these in order and write down what happens before moving on.

  1. Two clear blobs. Fifteen of each, well apart. It should find the boundary immediately, with everything correct.
  2. Overlapping blobs. Push them together until they mix. Some points end up wrong, and they should be the ones in the middle. The uncertainty band widens, which is the model being honest.
  3. One class surrounding the other. A blob of A in the center, a ring of B around it. Try to get above chance. You cannot, and the failure is total rather than partial.
  4. Two diagonal stripes. A stripe of A, then B, then A again. Same outcome.

What experiments 3 and 4 show

The boundary is a straight line. There is no straight line with the inside of a ring on one side and the outside on the other, so no amount of training helps. The model is not undertrained or badly tuned. The shape it can express does not include the answer.

That distinction is worth holding on to. A model can fail because it has not learned enough, or because it could never represent the answer. The first is fixed with more data or more training. The second is only fixed by changing the model.

You have now seen the second kind, deliberately. Section 7 opens with exactly this failure and builds the thing that fixes it.

One trick that does work

Before moving on, try this on the ring. Add a third feature computed from the two you have.

const withRadius = drawn.map(p => ({
  x: [p.x[0], p.x[1], (p.x[0] - 5) ** 2 + (p.x[1] - 5) ** 2],
  y: p.y
}));
model3 = trainLogistic(withRadius, 0.05, 40000);

Now it works. The new feature is the squared distance from the middle, and in that three dimensional space the two classes really are separated by a flat boundary. The line was never the problem, the coordinates were.

This is called feature engineering, and for decades it was most of the job. You looked at the data, worked out what transformation would make it linearly separable, and wrote it by hand. What neural networks changed is that they find those transformations themselves.

Exercises

  1. Draw a case where accuracy is high and recall is poor. Take a screenshot of the arrangement.
  2. Place one point of class B deep inside the A blob. How much does one point move the boundary?
  3. For the diagonal stripes, invent a feature that makes them separable. Hint: something periodic.
  4. Add an undo button. You will want it.
PreviousNext