Try first

You have a training loop and you have a map of the landscape. Before you draw anything, predict the path. Starting from the bottom left corner of the map, what shape does the walker trace on its way to the minimum? A straight line? A curve? Sketch it.

Record the path and draw it

function trainPath(data, lr = 0.01, steps = 5000) {
  let m = 0, b = 0;
  const path = [[m, b]];
  for (let i = 0; i < steps; i++) {
    const [dm, db] = gradient(m, b, data);
    m -= lr * dm;
    b -= lr * db;
    path.push([m, b]);
  }
  return path;
}

function showPath(path, mMin, mMax, bMin, bMax) {
  lossMap(mMin, mMax, bMin, bMax);
  view.xMin = mMin; view.xMax = mMax;
  view.yMin = bMin; view.yMax = bMax;
  for (let i = 1; i < path.length; i++) {
    line(path[i-1][0], path[i-1][1], path[i][0], path[i][1], "#fff");
  }
  dot(path[0][0], path[0][1], "#fff", 4);
  dot(path[path.length-1][0], path[path.length-1][1], "#000", 4);
}

showPath(trainPath(study), 0, 16, 0, 60);

trainPath is the training loop with one extra line, keeping every position it visited. showPath draws the map underneath and joins the positions with white segments. White dot is the start, black dot is the end.

What the path shows

Three things, and none of them are visible from the numbers alone.

It dives, then crawls. The first segment covers a large part of the picture. Thousands of later steps are packed into a short stretch near the end and you cannot tell them apart. This is the steep-then-flat loss curve from the last lesson, drawn in position rather than in loss.

It does not head for the target. The walker aims at the nearest lower ground, reaches the valley floor quickly, and only then turns and works along the floor towards the minimum. It travels a longer route than a straight line, because at no point does it know where it is going. This is exactly what the arrows in lesson 3055 predicted.

The turn is sharp. The path bends where it hits the valley, because the steep direction changes from across-the-valley to along-it in a very short distance.

Now make it bounce

showPath(trainPath(study, 0.035), 0, 16, 0, 60);

With a larger learning rate the path develops a visible zigzag. Each step overshoots the valley floor and lands on the opposite slope, so the walker crosses back and forth while inching forward.

This is the consequence of the diagonal valley that lesson 3050 said to expect. The valley is steep across and shallow along. One learning rate has to serve both. Big enough to make progress along the floor is too big to sit still across it.

Try 0.039, and then 0.04. Somewhere in there the bouncing stops being an inconvenience and starts diverging. The next lesson is about that boundary.

0 8 16 0 20 40 60 m b learning rate 0.01 0 8 16 0 20 40 60 m learning rate 0.035 ring is the start, filled dot is where it ended up
Both runs start at m = 0, b = 0 and end in the same place. The bend is where the steepest direction stops pointing across the valley and starts pointing along it.

What to watch

The path is drawn on a map that only exists because there are two parameters. With three you could not draw it, and everything above would still be true. Later, when a network trains badly, you cannot look at the picture. You infer the shape from the loss curve, and the shapes you are learning to recognize here are the ones you will be inferring.

Exercises

  1. Draw paths from four different starting corners on one map. Do they merge before reaching the minimum?
  2. Plot loss against step number on a separate canvas. Match each feature of that curve to a feature of the path.
  3. Draw only every fiftieth position as a dot rather than a line. Where do the dots bunch up, and what does the spacing measure?
  4. Run at lr = 0.001 for 5000 steps and draw it. Does the path reach the minimum? Say what went wrong before reading the next lesson.
PreviousNext