Try this first

Measuring a slope costs two evaluations of the loss, and every evaluation walks the whole dataset. For a large model that is the expensive part.

Question: is there a way to get the slope without any probing at all? Think about what you already know about the loss. You wrote it yourself, so you know its exact formula.

You can work it out once, on paper, forever

The loss is

L = (1/N) * sum of e * e     where   e = y - (m * x + b)

We want to know how L changes when m changes. Do it in two steps.

First, how does e * e change when e changes? Squaring something has slope 2e. If e is 3, nudging it up by a little raises e * e by about six times that little.

Second, how does e change when m changes? Look at the formula. m is multiplied by x and then subtracted, so raising m by a little lowers e by x times that little. The rate is -x.

Now multiply the two rates together. If moving m moves e at rate -x, and moving e moves the squared term at rate 2e, then moving m moves the squared term at rate 2e * (-x). Average over all the points and you have it.

dL/dm = -(2/N) * sum of (e * x)
dL/db = -(2/N) * sum of e

The second one comes out the same way. b is added directly, so raising it by a little lowers e by exactly that little, giving a rate of -1.

Multiplying rates along a chain like that is called the chain rule, and it is the single idea that makes neural networks trainable. Section 8 does nothing else. You have now used it twice.

m the slope b the intercept e = y − (mx + b) e ² the loss rate −x rate −1 rate 2e dL/dm = 2e · (−x) dL/db = 2e · (−1) follow a route, multiply the rates you pass
A rate of a rate. Each arrow answers one small question, and the answers multiply. This is the whole of the chain rule, and section 8 does nothing else.

In code

function gradient(m, b, data) {
  let dm = 0, db = 0;
  for (const r of data) {
    const x = r.x[0];
    const e = r.y - predict(m, b, x);
    dm += e * x;
    db += e;
  }
  return [-2 * dm / data.length, -2 * db / data.length];
}

One pass over the data produces both slopes. Compare that to probing, which needed two passes per parameter. With a million parameters the difference is the difference between possible and impossible.

Check it against the slow version

Never trust a derivative you worked out by hand until you have compared it to a measured one.

const m0 = 5, b0 = 20;
const lossM = m => loss(m, b0, study);
const lossB = b => loss(m0, b, study);

log("worked out", gradient(m0, b0, study).map(v => v.toFixed(5)).join("  "));
log("measured  ", slopeAt(lossM, m0).toFixed(5), " ", slopeAt(lossB, b0).toFixed(5));

The two lines should agree to about five decimal places. If they do not, the formula is wrong, and it is always the formula rather than the measurement. This check is called gradient checking and it comes back properly in lesson 3091.

What to watch

Look at what dm is made of: the residual multiplied by the input. A point with a large error pulls harder. A point with a large x also pulls harder, whether or not its error is large. That second part is why features on wildly different scales make training difficult, and why lesson 3042 exists.

Exercises

  1. Run the check at three different (m, b) pairs, including one far from the minimum. Confirm the agreement holds everywhere.
  2. Deliberately break gradient by dropping the minus sign, and watch the check fail. Remember what that failure looks like.
  3. Work out by hand what both slopes are at the exact minimum, without computing anything. What must they be, and why?
  4. Redo the derivation for a loss that uses Math.abs(e) instead of e * e. Where does it break down?
PreviousNext