Try first

You train the digit network. After 30 epochs the accuracy is 10 percent, which is what guessing would give.

Write down, in order, the first five things you would check. The order matters more than the list, and this lesson is mostly about the order.

Check the cheap things first

The instinct is to reach for the learning rate. Resist it. Start with the checks that take seconds and rule out whole categories.

1. Is the loss at the right starting value?

const fresh = makeMLP([256, 48, 10]);
log("start loss", meanLoss(fresh, s.train).toFixed(4), " expected", Math.log(10).toFixed(4));

With ten classes and random weights, every probability should be about 0.1, giving a cross entropy of 2.303. If you see that, the output layer, the softmax and the loss are all wired correctly.

If you see 0, the model is already certain about something and the softmax is probably broken. If you see 15, the initial weights are far too large. If you see NaN, there is a log of zero somewhere.

One line, and it clears three possible faults at once.

2. Can it memorize ten examples?

const ten = s.train.slice(0, 10);
const probe = makeMLP([256, 48, 10]);
trainMLP(probe, ten, 300, 0.1, 10);
log("accuracy on ten", accuracy(probe, ten));

This is the single most useful test in the list. Ten examples, no regularization, train until it fits. It should reach 100 percent, because a 12800 parameter model memorizing ten images is trivial.

If it cannot, the problem is not overfitting, capacity, or data volume. Something is broken. The gradient, the update, or the wiring. Fix that before looking at anything else.

If it can, the learning machinery works and your problem is data, capacity or hyperparameters. You have just cut the search space in half.

3. Do the labels match the data?

for (let i = 0; i < 10; i++) {
  drawDigit(s.train[i].x, i * 26 + 4, 4, 1.5);
}
log(s.train.slice(0, 10).map(r => r.y).join(" "));

Look at the pictures and read the labels underneath. If they do not match, nothing downstream can work, and no amount of tuning will tell you why.

A shuffle applied to the features but not the labels is the classic version of this, and it produces exactly the symptom in the opening: a model that trains smoothly and predicts at chance.

4. Check the gradient

Run gradCheck from lesson 3091 on a small network. If the relative error is above 1e-4, the backward pass is wrong and everything else is noise.

Do this whenever you have written or modified gradient code, including adding a layer type or changing an activation.

5. Sweep the learning rate

for (const lr of [1e-4, 1e-3, 1e-2, 1e-1, 1, 10]) {
  const net = makeMLP([256, 48, 10]);
  trainMLP(net, s.train, 3, lr, 16);
  log("lr", lr, " loss", meanLoss(net, s.train).toFixed(4));
}

Three epochs each, powers of ten, as in lesson 3059. You are looking for the range where the loss moves at all. Only now is it worth tuning.

What each symptom usually means

loss does not move at all        lr far too small, or gradient is zero
loss becomes NaN                 lr too high, or log of zero
loss falls then flattens high    underfitting, or a plateau
train falls, val rises           overfitting, section 9
accuracy stuck at 1/k            labels misaligned, or output layer broken
loss falls but predictions all
  one class                      class imbalance, or a collapsed layer

The last one is worth knowing. If 90 percent of your data is one class, predicting that class always is a decent local minimum and the model may settle there. Print the distribution of predictions, not only the accuracy.

what you see what it usually is the loss never moves at all the learning rate, or a gradient of zero the loss climbs smoothly the learning rate is too high the loss is NaN from step one a log of zero, or a divide by zero it cannot memorize ten rows the model or the gradient is wrong it memorizes ten but not the set not enough capacity, or bad features training good, unseen data bad overfitting, so go back to section 9 predicts at chance, trains smoothly the labels do not line up with the rows a model that will not learn is almost never a subtle problem
Check the cheap things first, and in this order. Most of a day lost to debugging is spent on the fifth row when the answer was in the first.

The general habit

Change one thing at a time. Two changes and a fix tell you nothing about which one worked.

Keep a log. Date, what you changed, what the numbers did. After twenty experiments you will not remember, and you will repeat one.

Make the loop fast. If an experiment takes ten minutes you will run six an hour. Cut the dataset to a tenth and the epochs to a fifth while debugging: you are looking for signs of life, not for a final number.

Get it working badly, then improve it. A model that runs end to end at 40 percent is far more valuable than a perfect architecture that has never executed.

What to watch

The hardest failures are the ones where nothing is broken. The code is correct, the gradient checks out, the loss falls, and the model is still not good enough. At that point the answer is more data, different features, or a different problem, and no amount of debugging will produce it. Recognizing that point early saves weeks.

Exercises

  1. Break something on purpose: shuffle the labels only. Work through the checklist and see which step catches it.
  2. Do the same with a sign flipped in the update rule. Which step catches that one?
  3. Write sanityCheck(net, data) that runs steps 1, 2 and 3 and prints a verdict for each.
  4. Train on a version of the data where one class is 90 percent of the rows. Report accuracy and the distribution of predictions.
PreviousNext