Try first

You split your data in two: train on one half, measure on the other. You try twelve models and keep the one with the best score on the held out half.

Is that held out score an honest estimate of how the winner will perform on new data? Think carefully before answering.

No, and the reason is subtle

You used the held out half twelve times, to make a choice. The winner is the model that happened to do best on those particular rows, which means part of its advantage is luck specific to that half.

Nothing was trained on it, so this is not the obvious kind of cheating. But information flowed from those rows into your decision, and the score is now optimiztic. Try enough models and you can get a good score on any fixed set by chance alone.

Hence three splits, not two.

training     60%   fit the parameters
validation   20%   choose between models and settings
test         20%   measure the winner, once

Training is what gradient descent sees. Validation is for every decision you make: how many layers, which learning rate, when to stop, which threshold. Test is looked at once, at the very end, to report a number.

The rule for the test set is strict and it is the whole point. Look at it once. If you look, adjust something, and look again, it has become a validation set and you no longer have an honest estimate.

Splitting properly

function shuffled(data) {
  const a = data.slice();
  for (let i = a.length - 1; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [a[i], a[j]] = [a[j], a[i]];
  }
  return a;
}

function split(data, trainFrac = 0.6, valFrac = 0.2) {
  const a = shuffled(data);
  const nTrain = Math.floor(a.length * trainFrac);
  const nVal = Math.floor(a.length * valFrac);
  return {
    train: a.slice(0, nTrain),
    val: a.slice(nTrain, nTrain + nVal),
    test: a.slice(nTrain + nVal),
  };
}

Shuffle first, always. Data arrives sorted more often than people expect: by date, by class, by whoever collected it. Split a sorted file at 60 percent and your training set may contain no examples of the last class at all.

training 60% validation 20% test 20% fit on training, choose settings on validation, look at test once when there is not enough data: five folds round 1 round 2 round 3 round 4 round 5 every row trains on four fifths and is judged on the fifth it never saw
Validation is what you tune against. Test is looked at once, at the end. Every time you make a decision using the test set, it quietly stops being a test set.

Three ways people leak the answer

Scaling before splitting. Fit minMaxScaler on all the data and its minimum and maximum have seen the test rows. Those numbers then influence training. Fit the scaler on the training set alone and apply it to the others. This is the promise lesson 3042 made, and it is the most common leak there is.

const s = split(rows);
const f = minMaxScaler(s.train);            // fitted on train only
const tr = s.train.map(r => ({ x: f(r.x), y: r.y }));
const va = s.val.map(r   => ({ x: f(r.x), y: r.y }));
const te = s.test.map(r  => ({ x: f(r.x), y: r.y }));

Duplicates across the split. If the same row, or a near copy, is in both training and test, the test score is measuring memory. Deduplicate before splitting.

Time. For anything ordered in time, a random split lets the model train on the future and predict the past. Split by date instead: train on everything before a cut off, test on everything after. It is a harder test and it is the one that matches how the model will be used.

When you do not have enough data

Holding back 40 percent of 50 rows leaves too little to train on and too few to measure with. Cross validation reuses everything.

function crossValidate(data, k, fit, score) {
  const a = shuffled(data);
  const size = Math.floor(a.length / k);
  const results = [];
  for (let i = 0; i < k; i++) {
    const hold = a.slice(i * size, (i + 1) * size);
    const rest = a.slice(0, i * size).concat(a.slice((i + 1) * size));
    results.push(score(fit(rest), hold));
  }
  return results;
}

Split into k parts. Train k times, each time holding out a different part. Every row is used for training k-1 times and for measuring once.

You get k scores instead of one, and their spread is useful in itself. A mean of 0.85 from scores of 0.84, 0.85, 0.86 means something different from the same mean from 0.61, 0.92, 0.99. The cost is training k times, which is why it is used on small datasets and rarely on large models.

What to watch

The test set answers one question: how well does this model do on data like this. It says nothing about data unlike this. A model tested on last year’s customers and deployed on next year’s has a test score that describes a situation that no longer exists. That kind of drift is not detectable by any split.

Exercises

  1. Split the house data from lesson 3062 into 6, 2 and 2. Train, and report all three errors.
  2. Deliberately fit the scaler before splitting and measure the difference. Is it large enough to notice?
  3. Run 5-fold cross validation on the polynomial fit for degrees 1 to 9. Compare the chosen degree to lesson 3096.
  4. Build a dataset where the label is correlated with row order, split without shuffling, and report the damage.
PreviousNext