Try first

You have a million training rows. Your training loop computes the gradient over all of them, then takes one step.

How many rows does it read before the model changes at all? And is reading all of them actually necessary to work out roughly which way is downhill?

You do not need every row to find the direction

A million rows for one step is enormous waste. And the gradient is an average, so a sample of it is an estimate of it. Take 32 rows at random and you get a direction that is not exactly right but is close, for a thirty thousandth of the cost.

Thirty thousand slightly wrong steps beat one exactly right step, easily. That is the whole argument.

Three words

batch        the rows used for one gradient computation
iteration    one batch, one gradient, one update
epoch        enough iterations to have seen every row once

With 1000 rows and a batch size of 50: 20 iterations per epoch, 20 updates per pass through the data.

Three variants, by batch size:

batch size = all rows    full batch, what you have been doing
batch size = 1           stochastic, one row at a time
batch size = 32 or so    mini-batch, what everyone actually uses

The middle one is called stochastic gradient descent, and in practice the name is used for the third as well.

The code

function trainSGD(net, data, lr = 0.5, epochs = 200, batchSize = 8) {
  const history = [];
  for (let e = 0; e < epochs; e++) {
    const shuffledData = shuffled(data);
    for (let i = 0; i < shuffledData.length; i += batchSize) {
      const batch = shuffledData.slice(i, i + batchSize);
      step(net, backwardAll(net, batch), lr);
    }
    history.push(netLoss(net, data));
  }
  return history;
}

Two additions to the old loop. Shuffle at the start of every epoch, and take the gradient from a slice rather than the whole set.

The shuffle matters. Without it, every epoch sees the same batches in the same order, so the model repeats the same sequence of nudges and can settle into a rhythm rather than converging.

What the loss curve looks like

for (const bs of [1, 8, 32, 1000]) {
  const net = makeNetwork(2, 8);
  const h = trainSGD(net, bigData, 0.5, 100, bs);
  log("batch", String(bs).padStart(4),
      " final", h[h.length-1].toFixed(4),
      " last 10 spread", (Math.max(...h.slice(-10)) - Math.min(...h.slice(-10))).toFixed(4));
}

Full batch gives a smooth curve that falls steadily. Small batches give a noisy curve that falls faster overall and jitters throughout, and never quite settles.

The jitter is not a defect. Each batch is a slightly different estimate of the gradient, so the model is nudged around rather than sliding cleanly downhill.

Why the noise helps

Lesson 3060 listed adding noise as a way to escape shallow local minima, and this is where it comes from for free.

A full batch gradient at a local minimum is exactly zero, so the model stops. A mini-batch gradient at the same point is not zero, because that batch has its own slightly different minimum. The model keeps moving and can wander out of a shallow dip. It cannot wander out of a deep one, which is what you want.

So mini-batching is faster and finds better answers. It is one of the few places in this subject where the cheap option is also the better one.

Choosing the batch size

Between 32 and 512 covers most of what people use, and here is what pulls in each direction.

Smaller means more updates per epoch, more noise, better escape from local minima, and worse use of hardware. Batches are processed in parallel, so a batch of 1 leaves most of a GPU idle.

Larger means a more accurate gradient, smoother progress, better hardware use, and a real risk of settling into the first minimum it finds.

Batch size and learning rate are linked. A larger batch gives a more reliable gradient, so you can safely take a larger step. The common rule of thumb is to scale the learning rate with the batch size, so doubling one means doubling the other.

What to watch

With mini-batches the loss you print during training is measured on one batch, so it bounces around for reasons that have nothing to do with the model improving. Do not judge progress from it. Compute the loss on the full training set, or on the validation set, once per epoch, and judge from that.

Exercises

  1. Plot loss against wall clock time, not epochs, for batch sizes 1, 8 and 1000. Which wins?
  2. Remove the shuffle and train for 200 epochs. Describe what the loss curve does.
  3. Halve the batch size and halve the learning rate. Is the result closer to the original than changing only one?
  4. Run the XOR experiment from lesson 3093 with batch size 1. Does the success rate over 20 runs improve?
PreviousNext