Try first

A test for a disease that affects 1 person in 1000. Here is a model for it.

function diagnose(patient) { return "healthy"; }

What is its accuracy? Is it a good test? Answer both before reading on.

Accuracy hides everything that matters

It is 99.9 percent accurate and completely worthless. It never finds a single case, which is the only thing anyone wanted it for.

That is the problem with accuracy on unbalanced data. When one class is rare, always guessing the common one scores well, and the score tells you nothing about whether the rare case is ever caught.

Accuracy also treats both mistakes as the same. Telling a healthy person they may be ill costs an anxious week and another test. Telling an ill person they are fine can cost them their life. One number cannot express that and should not try.

Four boxes, not one number

Split the mistakes by which kind they are.

                  predicted 1     predicted 0
actually 1     true positive    false negative
actually 0     false positive   true negative
function confusion(w, b, data, threshold = 0.5) {
  const c = { tp: 0, fp: 0, tn: 0, fn: 0 };
  for (const r of data) {
    const pred = classify(w, b, r.x, threshold);
    if (r.y === 1 && pred === 1) c.tp++;
    else if (r.y === 1 && pred === 0) c.fn++;
    else if (r.y === 0 && pred === 1) c.fp++;
    else c.tn++;
  }
  return c;
}

Every prediction lands in exactly one box. Nothing is lost, unlike accuracy, which adds the diagonal together and discards the rest.

Precision and recall

function scores(c) {
  const precision = c.tp / (c.tp + c.fp || 1);
  const recall    = c.tp / (c.tp + c.fn || 1);
  const f1 = 2 * precision * recall / (precision + recall || 1);
  return { precision, recall, f1 };
}

Precision: of everything the model flagged, how much was real. It is the number you care about when acting on a positive is expensive. A spam filter with poor precision deletes real mail.

Recall: of everything that was real, how much did the model find. It is the number you care about when missing a positive is expensive. A cancer screen with poor recall sends sick people home.

Run both on the always-healthy model. Recall is 0, because it found none of the sick. Precision is undefined, because it flagged nobody. Accuracy said 99.9 percent. These two say what accuracy would not.

The threshold is a dial between them

Precision and recall pull against each other, and the threshold is what trades one for the other.

for (const t of [0.1, 0.3, 0.5, 0.7, 0.9]) {
  const s = scores(confusion(w, b, points, t));
  log("t", t, " precision", s.precision.toFixed(3),
      " recall", s.recall.toFixed(3), " f1", s.f1.toFixed(3));
}

Lower the threshold and the model flags more things. Recall goes up because it catches more of the real cases, and precision goes down because more of what it flagged is wrong. Raise the threshold and the trade reverses.

At the extremes it is obvious. Threshold 0 flags everything, giving perfect recall and terrible precision. Threshold 1 flags nothing.

Nothing in training chose 0.5. It is the point where the model thinks the two classes are equally likely, which has nothing to do with what the two mistakes cost you. Pick the threshold from the cost, after training, and pick it on data the model has not seen.

F1, and when not to use it

F1 combines the two into one number, using a mean that punishes the smaller of the pair. That means you cannot win by pushing one to 1 and ignoring the other.

It is convenient for comparing models and it is a poor way to make a decision, because it assumes the two mistakes cost the same. They almost never do. If you know the real costs, use them directly.

What to watch

These are all measured against a threshold, so they all move when you move it. Comparing your precision to someone else’s precision at an unstated threshold is meaningless. Quote the threshold, or quote a threshold-free measure such as the area under the ROC curve.

Exercises

  1. Build a dataset where class 1 is 5 percent of the rows. Train, then report accuracy, precision and recall. Which one flatters the model?
  2. Sweep the threshold from 0 to 1 in steps of 0.02 and plot precision against recall. Where is the knee?
  3. Suppose a false negative costs 100 and a false positive costs 3. Write a function that picks the threshold minimizing total cost.
  4. Write down which of precision or recall matters more for: fraud detection, a search engine, a smoke alarm, a resume filter. Justify each in one line.
PreviousNext