Try first

Run training at three learning rates and write down the final loss for each.

for (const lr of [0.0001, 0.01, 0.05]) {
  const [m, b] = train(study, lr, 5000);
  log(lr, " m", m.toFixed(3), " b", b.toFixed(3), " loss", loss(m, b, study).toFixed(4));
}

Three very different outcomes. Explain each one before reading on.

Three failure modes and one success

Too small. At 0.0001 the answer is not wrong, just unfinished. Every step goes the right way and covers almost no ground. Five thousand steps were not enough to arrive. Give it five hundred thousand and it gets there.

About right. At 0.01 the loss lands near its minimum and the parameters are sensible.

Too large. At 0.05 you get NaN. The numbers start growing on the first step and pass what a double can hold at around step 1400.

Why too large gives NaN rather than a bad answer

Follow one step. You are on a slope, so the gradient is large. A large learning rate multiplies it into a step that overshoots the valley and lands higher up the far side than where you started. That new position is steeper, so the next gradient is larger, so the next step is longer still.

Each step feeds the next. The numbers grow by roughly the same factor every step. By step 24 m is already about 2.45e6, and somewhere near step 1400 they pass what a double can hold. JavaScript gives Infinity, then Infinity - Infinity gives NaN, and from then on every number in the model is NaN.

Watch it happen.

let m = 0, b = 0;
for (let i = 0; i < 25; i++) {
  const [dm, db] = gradient(m, b, study);
  m -= 0.05 * dm;
  b -= 0.05 * db;
  log(i, m.toExponential(3), b.toExponential(3));
}

The exponent climbs by roughly a fixed amount each line. That steady climb is the signature of divergence, and it is worth being able to recognize on sight. A loss that grows smoothly means the learning rate is too high. A loss that jumps to NaN in one step usually means something else, often a division by zero or a log of zero, which you meet in section 6.

Finding a rate without guessing

The usual method is to sweep and look.

for (let e = -5; e <= -1; e += 0.5) {
  const lr = Math.pow(10, e);
  const [m, b] = train(study, lr, 500);
  const l = loss(m, b, study);
  log(lr.toExponential(1), Number.isFinite(l) ? l.toFixed(4) : "diverged");
}

Try rates spaced by powers of ten, not by even amounts. The useful range spans several orders of magnitude, so stepping by 0.001 at a time would waste most of your attempts in one narrow band.

Take the largest rate that has not diverged, then back off by a factor of two or three. Sitting right at the edge trains fastest and breaks as soon as anything changes.

The thing that makes this much easier

The safe range depends on the size of your features. Bigger inputs give bigger gradients, which need a smaller rate. Scale your features to a similar range, as in lesson 3042, and a rate near 0.01 tends to work across quite different problems. Skip the scaling and you tune the rate from scratch every time.

That is the practical reason feature scaling is standard advice, and it is a better reason than the one about distances.

What to watch

The learning rate is not a parameter. Training does not adjust it, because it is not part of the model. It is a hyperparameter: a setting you choose that controls how learning happens. Model architecture, batch size, and the regularization strength in lesson 3098 are all hyperparameters too, and none of them are found by gradient descent.

Exercises

1. Find the highest learning rate that still converges, to two significant figures. Compare it to the 0.035 that zigzagged in the last lesson.
2. Scale the study data with minMaxScaler, retrain, and find the highest safe rate again. How much did it change?
3. Implement a schedule that starts at 0.03 and multiplies the rate by 0.999 each step. Does it beat a fixed rate for the same number of steps?
4. Add a guard that halves the learning rate whenever the loss rises. Run it at a rate you know diverges and see whether it recovers.