Try this first

You did exercise 2 of the last lesson, so you have the totals. Here is the point of it, made worse.

A line predicts 100 for a student who scored 20, and predicts 20 for a student who scored 100. Add the two residuals. What total do you get, and what does that total claim about the line?

Adding residuals is useless

The residuals are -80 and +80, so the total is zero. By that measure the line is perfect, while being as wrong as it is possible to be on both points.

The problem is that a residual carries a direction, and directions cancel. Any measure of badness has to treat “too high by 5” and “too low by 5” as equally bad. So we need to strip the sign off before adding.

There are two sensible ways to do that.

The two options

Math.abs(e)   // size, sign discarded
e * e         // square, always positive

Both work. Both give zero only when the residual is zero. Both grow as the error grows. You could build a whole course on either. Here is what actually separates them.

Squaring punishes large errors much harder. Two errors of 5 give 25 + 25 = 50 when squared. One error of 10 gives 100. To the absolute version those two situations are identical, both totalling 10. Squaring says one big miss is worse than several small ones, which is usually what people mean by a bad fit.

Squaring is smooth. This is the reason that decides it. Plot both functions.

view.xMin = -3; view.xMax = 3;
view.yMin = 0; view.yMax = 9;
clear(); grid(1);
for (let e = -3; e <= 3; e += 0.05) {
  dot(e, e * e, "#38a", 2);
  dot(e, Math.abs(e), "#c33", 2);
}

The blue curve is a smooth bowl. The red one is a V with a sharp corner at zero. In section 4 we find the best line by asking, at every step, which way is downhill. At the corner of the V there is no single answer to that question. The bowl has a well defined slope everywhere, including at the bottom.

Being honest about the trade

Squaring has a real cost. Because a big error is punished so hard, a single wrong data point can drag the whole line towards itself. One student recorded as 8 hours and 3 marks, by a typo, will bend the line more than the other seven points together.

The absolute version is far more resistant to that, and it has a name, mean absolute error. It is a good choice when your data has outliers you cannot remove. It is used less often mostly because of the corner, and because squared error has clean mathematics behind it.

We use squared error for the rest of this course. That is a choice with consequences, and it is worth knowing you made it.

What to watch

Squaring changes the units. Residuals are in marks. Squared residuals are in marks squared, which is not a quantity anyone has intuition about. A loss of 24 does not mean the line is 24 marks off. Take the square root at the end and you get back to marks, which is what people mean by root mean squared error.

Exercises

  1. For residuals [5, 5] and [10, 0], compute both totals under both methods. Confirm the claim above with your own numbers.
  2. Take the study data, change one score to an obvious typo, and see how far you have to move the line by eye before it looks reasonable again.
  3. The corner in the V is at zero. Explain why a search that asks “which way is downhill” would get stuck there, in your own words. Then check your answer against lesson 5 of section 4.
  4. Suggest a measure that behaves like squaring for small errors and like absolute value for large ones. Such a thing exists and is called Huber loss. Try to invent it before looking it up.
PreviousNext