Try first

You are at m = 12. The slope dL/dm comes back as +40.

Should m go up or down? Answer, and say why in one sentence, before reading on.

The gradient points uphill, so we go the other way

A positive slope means increasing m increases the loss. We want the loss smaller, so m must come down. If the slope had been negative, increasing m would lower the loss, and m should go up.

In both cases you move against the sign of the slope. That is what subtracting does.

m = m - learningRate * dm
b = b - learningRate * db

This is the update rule. Every training algorithm in this course is a variation on these two lines, and so is most of what is running in a data center right now.

Reading it piece by piece

The minus sign turns uphill into downhill. Change it to a plus and the loss climbs, quickly and without any error message. This is a genuinely common bug and the symptom is a loss that grows every step.

The learning rate is how much of the slope you act on. The gradient gives a direction and a steepness, but nothing about it says how far to walk. That number is yours to choose, and the next two lessons are about choosing it.

The step is proportional to the slope, which is the quiet virtue of the rule. Far from the bottom the ground is steep, the gradient is large, and the step is large. Near the bottom the ground flattens, the gradient shrinks, and the steps shrink with it. The walker slows down as it arrives, without being told to.

Contrast that with the fixed step in lesson 3052, which could never land nearer than 0.25 to the answer no matter how long it ran.

0 4 8 12 16 0 1000 2000 m loss slope −, go right slope +, go left 0 8 16 m nine steps, same rule each time the gaps shrink on their own
The minus sign turns the slope into a direction. The size of the slope then supplies the step length for free, so the walker slows down as it arrives without being told to.

Work one step by hand

m = 12, b = 20,  learningRate = 0.01
gradient = [40, 8]

m = 12 - 0.01 * 40 = 11.6
b = 20 - 0.01 *  8 = 19.92

Both moved down. m moved five times further than b, matching the five to one ratio in the gradient. Nobody decided that. It fell out of the two slopes.

Notice both parameters update from the same gradient, the one measured before either moved. Update m first and then recompute the gradient for b, and you are no longer descending the surface you think you are. Compute the whole gradient, then apply the whole update.

The case where the rule has nothing to plug in

Back to the V from lesson 3047. With absolute error, the slope to the left of zero is -1 and the slope to the right is +1, and at zero itself there is no single answer. The update rule needs one number and the function offers two.

In practice people pick one by convention and carry on, and it mostly works. But it explains why the smooth bowl was worth choosing back when we had a free choice. Squaring gives the rule a well defined value everywhere, including at the point you are trying to reach.

-4 -2 0 2 4 e |e| slope −1 slope +1 no single slope here
The update rule needs one number and the V offers two. This is why the smooth bowl was worth choosing back when the choice was free.

What to watch

The learning rate has to suit the size of the gradients, which depend on the size of your data. The same 0.01 that works here would explode on features measured in thousands. That is a second reason to scale features, on top of the one in lesson 3042.

Exercises

  1. Do three updates by hand from (12, 20), recomputing the gradient each time with your gradient function. Is the loss falling?
  2. Flip the minus to a plus and do three more. Write down what the loss does.
  3. With a learning rate of 0.1 instead of 0.01, what is the new m after one step? Does that look safe?
  4. Write the update rule for a model with a hundred parameters held in an array. How many lines does it take? This is the answer to why the notation from lesson 3038 was worth building.
PreviousNext