Try this first

You have two slopes now, one for m and one for b. Each was worked out while holding the other still.

Question: if dL/dm is 40 and dL/db is 8, which way should you move in the (m, b) plane to go downhill fastest? Give a direction, not just a sign.

Two slopes make a direction

Put the two numbers together as a vector and you have one.

gradient = [dL/dm, dL/db] = [40, 8]

That vector points in the direction the loss climbs fastest, and its length says how steep the climb is. So downhill is the same vector with the sign flipped.

With 40 and 8, the steepest climb is mostly in the m direction and a bit in the b direction, in the ratio five to one. To descend fastest you move against that, changing m five times as much as b.

Each component answers its own narrow question: if I move only this parameter and freeze everything else, how does the loss respond? That is called a partial derivative. Collect all the partial derivatives into one vector and the collection is the gradient.

The word scales without changing meaning. Two parameters give a gradient of length two. Seventy billion parameters give a gradient of length seventy billion, still meaning the same thing: one number per parameter, saying which way and how hard that parameter pulls.

See it on the map

Draw the loss map from lesson 3050, then put arrows on it.

function arrows(mMin, mMax, bMin, bMax, n = 12) {
  view.xMin = mMin; view.xMax = mMax;
  view.yMin = bMin; view.yMax = bMax;
  for (let i = 0; i < n; i++) {
    for (let j = 0; j < n; j++) {
      const m = mMin + (i + 0.5) / n * (mMax - mMin);
      const b = bMin + (j + 0.5) / n * (bMax - bMin);
      const [dm, db] = gradient(m, b, study);
      const len = Math.sqrt(dm * dm + db * db) || 1;
      const sx = (mMax - mMin) / n * 0.4;
      const sy = (bMax - bMin) / n * 0.4;
      line(m, b, m - dm / len * sx, b - db / len * sy, "#fff");
    }
  }
}

lossMap(0, 16, 0, 60);
arrows(0, 16, 0, 60);

Every arrow is drawn with the gradient negated, so they all point downhill. Dividing by len makes them the same length, because otherwise the steep ones would run off the picture and the shallow ones would be invisible. We are drawing the direction only.

What the picture tells you

Every arrow points into the valley. Follow any of them and you get closer to the bottom, from anywhere on the map. That is the guarantee gradient descent is built on.

Look along the valley floor. The arrows there are almost perpendicular to the valley, pointing across it rather than along it. Steepest downhill is a local statement, and the local answer does not aim at the destination. It aims at the nearest bit of lower ground. In two lessons you will watch a walker zigzag across this valley for exactly that reason.

What to watch

The arrows all have the same length in the picture, which hides something important. The real gradient is enormous in the far corners and nearly zero at the bottom. That size difference is what makes the walker take large steps when far away and small ones when close, with no extra code. Print the raw length at a corner and at the middle to see the scale.

Exercises

1. Print the gradient at (0, 0) and at your best hand tuned point from lesson 3049. Compare the two lengths.
2. Remove the division by len and draw again. Describe what happens and why it makes the picture useless.
3. Along the valley floor, is the gradient ever exactly zero? Where, and how many such points are there?
4. Write gradientNumeric(m, b, data) using slopeAt twice, and draw the same arrows with it. The picture should be identical, which is gradient checking done visually.