Try this first

The line y = 8x + 24 predicts 56 for a student who studied 4 hours. The real score was 55.

How wrong is the line, at that point? Give a number, and be careful about the sign.

The residual

The answer is 1 if you write it as prediction minus actual, or -1 the other way round. Either is fine as long as you pick one and keep it. We will use actual minus predicted, so a positive value means the line sat too low.

error = y - (m * x + b)

That gap has a name. It is the residual: what is left over after the line has explained what it can. One residual per data point, so eight points give eight numbers.

function residuals(m, b, data) {
  return data.map(r => r.y - predict(m, b, r.x[0]));
}

log(JSON.stringify(residuals(8, 24, study).map(e => e.toFixed(1))));

Run it. You get a mix of positive and negative numbers, most of them small. That list is the complete record of how this line fails.

Draw them

Residuals are much easier to judge as a picture.

function showResiduals(m, b, data) {
  clear();
  grid(1);
  drawLine(m, b, "#333");
  for (const r of data) {
    const x = r.x[0];
    const p = predict(m, b, x);
    line(x, p, x, r.y, "#c33");
    dot(x, r.y, "#38a");
  }
}

showResiduals(8, 24, study);

Each red segment runs straight up or down from the line to the point. The length of the segment is the size of the residual, and now you can see which points the line handles well and which it does not.

Why vertical and not the shortest gap

The obvious objection: the shortest distance from a point to a line is perpendicular to it, so why measure straight up and down?

Because of what we are asking the model to do. Given a study time, predict a score. The study time is given and is not in question. The only thing that can be wrong is the score. So the mistake is measured along the score axis, and only along it.

The perpendicular distance would be measuring an error in the study time as well, and treating an error of one hour as comparable to an error of one mark. Those are different units, which is the same trap as lesson 5 of the last section.

0 2 4 6 8 20 60 100 hours score one bar per point, always vertical 4.6 5.0 5.4 64 68 hours the residual the shortest gap on the picture, and the wrong question one point, magnified
The residual is vertical because the question is "what score did we predict for this many hours". The perpendicular gap is shorter, and it answers a question nobody asked.

What to watch

Look at the pattern of the residuals, not only their sizes. If the line is a good description, the leftovers should look like scatter with no shape to them. If the residuals are all negative in the middle and positive at both ends, the data is curved and a straight line cannot fix it by moving. That is information you cannot get from a single summary number, which is why people plot residuals long after they have automated everything else.

Exercises

  1. Run showResiduals for all three candidate lines from lesson 1. Which one looks best, and can you say why in terms of the red segments?
  2. Add up the eight residuals for y = 8x + 24. Now do the same for y = 7.5x + 27. Does the total tell you which line is better? Do not read the next lesson until you have an opinion.
  3. Build a dataset from y = x * x for x from 1 to 8, fit any straight line to it by eye, and plot the residuals. Describe the shape you get.
  4. Change showResiduals to color positive residuals differently from negative ones. Then look at the curved dataset again.
PreviousNext