Try first

Three models. Their errors:

          train    fresh
model A    8.4      8.6
model B    0.9      1.1
model C    0.1      7.9

For each, say what is wrong and what you would do about it. Then read on.

Two ways to fail

Model A is underfitting. High error on both. It is too simple to capture what is in the data, and it fails the same way everywhere. Fitting a straight line to a curve gives this.

Model C is overfitting. Low training error, high fresh error. It has memorized this particular sample, noise included, and has nothing to say about anything else.

Model B is what you want. Both errors low and close together.

The gap between the two numbers is the diagnostic. A large gap means overfitting. Both high with a small gap means underfitting. There is no third pattern that matters, and no situation where fresh error is meaningfully lower than training error, so if you see that, suspect a bug in the split.

Where it comes from

Underfitting is a shortage of capacity. The model cannot express the pattern, and no amount of data or training fixes it. This is the ring in lesson 3077, and it is the same failure as XOR in lesson 3079.

Overfitting is an excess of capacity relative to data. The model has more freedom than the data can pin down, so the leftover freedom gets spent on noise.

Both are about the same ratio: how much the model can express, against how much the data constrains. Too little and you underfit, too much and you overfit.

Draw the curve

const trainErr = [], freshErr = [];
for (let d = 1; d <= 9; d++) {
  const m = fitPoly(sample, d);
  trainErr.push(rmseOn(m, sample));
  freshErr.push(rmseOn(m, fresh));
  log("degree", d, trainErr[d-1].toFixed(3), freshErr[d-1].toFixed(3));
}

view.xMin = 0.5; view.xMax = 9.5;
view.yMin = 0; view.yMax = Math.max(...freshErr) * 1.1;
clear(); grid(1);
for (let d = 1; d <= 9; d++) {
  dot(d, trainErr[d-1], "#38a", 4);
  dot(d, freshErr[d-1], "#c33", 4);
  if (d > 1) {
    line(d-1, trainErr[d-2], d, trainErr[d-1], "#38a");
    line(d-1, freshErr[d-2], d, freshErr[d-1], "#c33");
  }
}

Blue falls steadily and keeps falling. More capacity always fits the training data better, right down to zero, so training error alone can never tell you when to stop.

Red falls, reaches a minimum, then climbs. Left of the minimum you are underfitting, right of it you are overfitting, and the bottom of the red curve is the model you want.

That U shape is the single most useful picture in applied machine learning. Everything in this section is about finding its bottom.

1 2 3 4 5 6 7 8 9 0.0 0.4 0.8 degree of the polynomial error the model you want training error on unseen data underfitting on the left, overfitting on the right training error keeps falling the whole way, and stops being informative early
The gap between the two curves is overfitting, and the point where the upper one turns is the model you want. Training error alone would have told you to pick the worst one.

The three knobs

Model capacity. Polynomial degree, hidden units, layers. More capacity moves you right along the curve.

Amount of data. More data pushes the red curve down and to the right, so a bigger model becomes safe. This is the most reliable fix and usually the most expensive.

Regularization. Keep the large model and restrain it, which is lesson 3098.

Training time is a fourth, quieter one. Train a big model long enough and it drifts right along the curve on its own, which is why stopping early is a real technique.

What to watch

The modern picture is less tidy than the U suggests. Very large networks, with far more parameters than data, often generalize well past the point where this curve says they should collapse. The effect is real, it is called double descent, and it is not fully explained.

So treat the U as a reliable guide for models of the size you will build here, and know that at very large scale the story is more complicated than the textbook version.

Exercises

  1. Repeat the sweep with 100 sample points. Where does the red minimum move to?
  2. Repeat with the noise doubled. Which direction does the minimum move, and why?
  3. Train an 8 hidden unit network on 4 rows and on 400 rows. Report both gaps.
  4. Write diagnose(trainErr, freshErr) which returns “underfitting”, “overfitting” or “fine”, and say what thresholds you chose and why.
PreviousNext