Try first

The degree 9 model overfits. Suppose you must keep degree 9, and cannot get more data.

Look at its weights. What do you notice about their size compared to the degree 1 model? And can you use that?

Overfitted models have large weights

log("degree 1", m1.w.map(v => v.toFixed(1)).join(" "));
log("degree 9", m9.w.map(v => v.toFixed(1)).join(" "));
log("sizes   ", norm(m1.w).toFixed(1), norm(m9.w).toFixed(1));

The degree 9 weights are enormous, and they alternate in sign. That is what a wiggly curve requires: large opposing terms that mostly cancel, and where they fail to cancel exactly you get a swing.

So large weights and overfitting travel together. That gives us a handle. Penalize large weights in the loss and the model has to earn them.

L2 regularization

total loss = data loss + lambda * sum of (w * w)

Add the squared size of the weights to the thing being minimized. Now every weight has to justify itself: it will only grow if the reduction in data loss outweighs the penalty.

lambda sets the exchange rate. Zero gives you the old behavior. Very large drives every weight to zero, so the model predicts a constant. In between is a model that fits the strong patterns and ignores the weak ones.

The gradient is easy. The derivative of lambda * w * w is 2 * lambda * w, so add that to each weight’s existing gradient.

function gradientL2(w, b, data, lambda) {
  const [dw, db] = gradient(w, b, data);
  return [dw.map((v, j) => v + 2 * lambda * w[j]), db];
}

Two things worth noticing. The bias is not penalized, because it only shifts the output and does not make the model wigglier. And the extra term always points back towards zero, which is why this is also called weight decay: every step shrinks each weight a little, and the data has to push back to keep it.

Try it

for (const lambda of [0, 0.0001, 0.001, 0.01, 0.1, 1]) {
  const m = fitPolyL2(sample, 9, lambda);
  log("lambda", String(lambda).padEnd(8),
      " train", rmseOn(m, sample).toFixed(3),
      " fresh", rmseOn(m, fresh).toFixed(3),
      " |w|", norm(m.w).toFixed(1));
}

Read the three columns together. As lambda rises, training error rises steadily. Fresh error falls, reaches a minimum, then rises. The weight size falls throughout.

Draw the winning model and you get something close to the straight line, from a degree 9 polynomial. The capacity is still there and the model chose not to use it.

That is the useful idea. Rather than guessing the right model size, use a generous one and control it with a penalty you can tune continuously.

The other options

L1 penalizes Math.abs(w) instead of w * w. It pushes weights to exactly zero rather than merely small, so it selects features as well as shrinking them. Useful when you suspect most of your features are useless. It has the corner problem from lesson 3047.

Early stopping. Watch validation loss during training and stop when it starts rising. Costs nothing, needs no new hyperparameter, and is close to free. Almost everyone does this.

Dropout. During training, randomly switch off some fraction of the hidden units on each step. No unit can rely on any other being present, so the network cannot build fragile chains. Switch it off when predicting. Standard for large networks and it looks strange until you have seen it work.

More data. Always the best answer when available. Every technique here is a way of coping with not having enough.

What to watch

lambda is a hyperparameter, so it is chosen on the validation set, not by training and not on the test set. And its right value depends on how many rows you have: the data loss is an average and the penalty is not, so the same lambda restrains a small dataset far more than a large one.

Exercises

1. Write fitPolyL2, then plot fresh error against lambda on a log scale.
2. Draw the degree 9 curve at lambda 0, 0.001 and 0.1 on one canvas.
3. Implement early stopping for the XOR network with a validation split. When does it stop?
4. Add L2 to the network in lesson 3090. Which parameter groups should it apply to?