Try first

Ten measurements, and you fit a model that passes through every single one exactly. Training loss: zero.

Is that the best possible model? Answer before reading on.

Build the zero loss model

You can do this with the linear model you already have. Feed it powers of x as separate features and it fits a curve.

function polyFeatures(x, degree) {
  const f = [];
  for (let k = 1; k <= degree; k++) f.push(Math.pow(x, k));
  return f;
}

const truth = x => 2 + 1.5 * x;
const sample = [];
for (let i = 0; i < 10; i++) {
  const x = i * 0.9;
  sample.push({ x, y: truth(x) + (Math.random() - 0.5) * 2 });
}

function fitPoly(sample, degree, lr = 0.02, steps = 200000) {
  const rows = sample.map(s => ({ x: polyFeatures(s.x, degree), y: s.y }));
  const f = standardScaler(rows);
  const scaled = rows.map(r => ({ x: f(r.x), y: r.y }));
  const [w, b] = trainAll(scaled.map(r => r.x), scaled.map(r => r.y), lr, steps);
  return { w, b, f, degree };
}

function polyPredict(m, x) {
  return predict(m.w, m.b, m.f(polyFeatures(x, m.degree)));
}

The data is a straight line with noise added. We know that, because we made it. The model does not.

Look at what degree 9 does

const m1 = fitPoly(sample, 1);
const m9 = fitPoly(sample, 9);

view.xMin = -0.5; view.xMax = 9; view.yMin = -5; view.yMax = 20;
clear(); grid(1);
for (let x = -0.5; x <= 9; x += 0.02) {
  dot(x, polyPredict(m1, x), "#38a", 1);
  dot(x, polyPredict(m9, x), "#c33", 1);
}
for (const s of sample) dot(s.x, s.y, "#000", 4);

The blue line is a straight fit. It misses every point by a little.

The red curve threads through the points far more closely, and between them it swings up and down in ways nothing in the data suggests. Past the last point it leaves the picture entirely.

The red model has a lower training loss. It is much worse.

Test it on data it has not seen

The only honest test is a point that was not used to fit the model.

const fresh = [];
for (let i = 0; i < 10; i++) {
  const x = i * 0.9 + 0.45;
  fresh.push({ x, y: truth(x) + (Math.random() - 0.5) * 2 });
}

function rmseOn(m, data) {
  let s = 0;
  for (const r of data) {
    const e = r.y - polyPredict(m, r.x);
    s += e * e;
  }
  return Math.sqrt(s / data.length);
}

log("degree 1  train", rmseOn(m1, sample).toFixed(3), " fresh", rmseOn(m1, fresh).toFixed(3));
log("degree 9  train", rmseOn(m9, sample).toFixed(3), " fresh", rmseOn(m9, fresh).toFixed(3));

Degree 1 scores about the same on both. Degree 9 scores far better on the training rows and far worse on the fresh ones, often by a factor of ten or more.

What the model actually learned

The data is a line plus noise. The noise is random. It carries no information about anything, including about the next point.

The straight line model cannot represent the noise, so it ignored it and captured the line. The degree 9 model has enough freedom to represent both, so it fitted the line and the noise, and it cannot tell them apart. Every wiggle in the red curve is the model taking a random measurement error seriously.

That is overfitting: learning the accidents of your particular sample rather than the pattern behind it. And now the answer to the opening question. A training loss of zero is not a goal. It is a warning.

What to watch

The thing you actually want is called generalization: performing well on data you have not seen. Training loss does not measure it and cannot, because the model has already seen those answers.

Everything in this section follows from that one gap. You need a way to measure generalization, which is lesson 3097, and ways to encourage it, which are lessons 3098 and 3099.

Exercises

1. Fit degrees 1 through 9 and print train and fresh error for each. Where do the two part company?
2. Regenerate the sample with no noise at all. Does degree 9 still overfit? What does that tell you about the cause?
3. Increase the sample to 100 points and refit degree 9. How much does more data help?
4. Predict at x = 12 with both models. Which answer would you act on?