Try this first

You have a loss for every pair of numbers (m, b). That is a function of two inputs producing one output.

Before reading on: what kind of picture shows a function of two inputs? You have already seen one, on a weather map or a walking map.

The loss is a landscape

Put m along one axis and b along the other. Every point on that plane is one possible line. The loss at that point is the height of the ground there.

That gives you terrain. Bad lines are high ground. Good lines are low ground. The best line sits at the lowest point in the valley. Training is walking downhill.

This is the picture worth carrying for the rest of the course. It does not change when the model gets bigger. A network with a million parameters has a landscape in a million dimensions, which nobody can draw, but every statement about it is a statement about hills and valleys.

Draw it

function lossMap(mMin, mMax, bMin, bMax) {
  const w = canvas.width, h = canvas.height;
  const img = ctx.createImageData(w, h);

  let lo = Infinity, hi = -Infinity;
  const vals = new Float64Array(w * h);
  for (let py = 0; py < h; py++) {
    for (let px = 0; px < w; px++) {
      const m = mMin + (px / w) * (mMax - mMin);
      const b = bMax - (py / h) * (bMax - bMin);
      const v = Math.log(loss(m, b, study) + 1);
      vals[py * w + px] = v;
      if (v < lo) lo = v;
      if (v > hi) hi = v;
    }
  }

  for (let i = 0; i < w * h; i++) {
    const t = (vals[i] - lo) / (hi - lo);
    img.data[i * 4]     = Math.round(255 * t);
    img.data[i * 4 + 1] = Math.round(255 * (1 - Math.abs(t - 0.5) * 2));
    img.data[i * 4 + 2] = Math.round(255 * (1 - t));
    img.data[i * 4 + 3] = 255;
  }
  ctx.putImageData(img, 0, 0);
}

lossMap(0, 16, 0, 60);

The two loops walk every pixel. For each one they work out which (m, b) that pixel stands for, compute the loss there, and store it. Note the b line subtracts, because pixel rows count downwards while we want b to increase upwards.

The second loop turns each value into a color. t is 0 at the lowest loss and 1 at the highest, so blue is good and red is bad.

Math.log is there because the loss ranges over several orders of magnitude. Without it the whole picture is one flat color with a single dark speck, since the extremes swamp everything in between. Taking a log to make a picture readable is a very common move and it does not change where the minimum is.

What you are looking at

A long blue trough running diagonally, with the darkest point somewhere in the middle of it. Three things to take from the shape.

There is one basin. No separate pockets, no second valley. Wherever you start, downhill leads to the same place. Squared error on a straight line always gives this, and it is why linear regression is a solved problem.

The valley is a long diagonal, not a round bowl. That diagonal is what you felt in the last lesson when raising m and lowering b kept the loss about the same. Steeper lines that start lower fit nearly as well. In section 4 lesson 7 this shape is what makes a badly chosen step size bounce from side to side.

It is smooth. No cliffs, no steps. That comes directly from squaring rather than taking absolute values, and it is what lets us ask for a slope at any point.

What to watch

Computing this picture cost 172,800 evaluations of the loss, for two parameters. Add a third and the same approach needs the width of the picture times as many again. Grid searching a landscape stops being possible after about four parameters. Section 4 exists because of that count.

Exercises

  1. Mark your best hand tuned (m, b) from the last lesson on the map with a white dot. Is it in the darkest region?
  2. Remove the Math.log and look again. Describe what happened and why the log helped.
  3. Zoom in by calling lossMap(7, 9, 15, 35). The valley should look flatter, because you narrowed the range of losses on screen. What does that tell you about how much precision the last decimal place of m is worth?
  4. Change the data so that all eight points sit exactly on a line, then draw the map again. What happens to the lowest value, and what does the valley look like now?
PreviousNext