Try first

Train the linear model straight onto the 0 and 1 labels. Nothing new needed, it is the code from lesson 3067.

const X = points.map(p => p.x);
const y = points.map(p => p.y);
const [w, b] = trainAll(X, y, 0.05, 20000);

for (const p of [[1,1], [5,5], [9,9], [12,12]]) {
  log(p.join(","), predict(w, b, p).toFixed(3));
}

Look at the four outputs. Two of them are not probabilities. Say what went wrong before reading on.

A line does not stop

You get something like -0.3 for the bottom left corner and 1.4 for the far top right. A probability of -0.3 does not mean anything, and neither does 1.4.

This is not a bug you can patch. It is what a straight line is. The whole content of y = wx + b is that the output changes at a constant rate forever. Walk far enough in either direction and it passes any bound you like. There is no value of w and b that keeps a line inside 0 and 1 while still varying, except the flat line that always says the same thing.

Clamping does not fix it either

The obvious patch is to squash anything outside the range back in.

const clamped = Math.min(1, Math.max(0, predict(w, b, x)));

The output is now in range and the model is worse than before. Everything past the boundary is flattened to exactly 0 or exactly 1, and a flat region has zero slope. From lesson 3060 you know what zero slope does to gradient descent: every point that lands in the flat region contributes nothing to the gradient and cannot teach the model anything.

So the far away points, which are the ones the model got most confidently right or most confidently wrong, become invisible during training. That is the opposite of what you want.

What the fix has to look like

Write the requirements down and the answer nearly appears on its own.

• Output always between 0 and 1, for any input at all.
• Never exactly reaches either end, so the slope is never exactly zero.
• Increasing, so a larger score always means a higher probability.
• Smooth everywhere, so the chain rule works.
• Gives 0.5 at the point of maximum uncertainty, which we can put at input zero.

Sketch a curve satisfying all five. It has to be flat and near 0 on the far left, flat and near 1 on the far right, and rise through 0.5 in the middle. An S.

Keep the line, add a step

The linear part is not the problem and it is worth keeping. It combines features with weights, which is exactly what we want. The problem is only that its output is the wrong shape.

So do it in two stages.

score       = dotProduct(w, x) + b        // any number, -inf to +inf
probability = squash(score)               // between 0 and 1

The linear stage does the learning. The squashing stage does the shaping and has no parameters. Split that way, everything you built in sections 3 to 5 carries over unchanged, and the only new thing is one function.

That split is worth remembering, because it is exactly how a neuron is built in section 7. Linear part, then a fixed non-linear function. The whole of deep learning is that pair, repeated.

What to watch

The score is worth keeping around, not just the probability. It is the signed distance from the boundary, roughly speaking, so it tells you which side and how far. Lesson 3075 draws it.

Exercises

1. Find inputs to the trained linear model that give below -1 and above 2. How far out did you have to go?
2. Implement the clamped version and train it. Print the gradient after each step and find where it goes to zero.
3. Sketch three different S shaped curves meeting all five requirements. There is more than one answer, and two of them are in real use.
4. Explain why a flat line, always predicting 0.5, satisfies the range requirement and is still useless.