The whole thing, in one file

The converter from lesson one learned a single number. Now we learn two at once, and we structure the code so that each part has a name. This program is the skeleton of everything that follows.

Our data comes from the rule y = 2x + 1, but the program does not know that. It only sees the pairs.

const data = [[1, 3], [2, 5], [3, 7], [4, 9]];

let w = 0;               // weight
let b = 0;               // bias
const rate = 0.01;       // how far we move on each correction

function predict(x) {
  return w * x + b;
}

function loss() {
  let total = 0;
  for (const [x, y] of data) {
    const error = y - predict(x);
    total += error * error;
  }
  return total / data.length;
}

for (let epoch = 0; epoch < 1000; epoch++) {
  for (const [x, y] of data) {
    const error = y - predict(x);
    w += rate * error * x;
    b += rate * error;
  }
  if (epoch % 200 === 0) {
    console.log(epoch, loss().toFixed(5), w.toFixed(3), b.toFixed(3));
  }
}

Run it. The last line should print values very close to w = 2 and b = 1, and a loss very close to zero.

Reading it line by line

  • w and b are the only things that change. Everything else is fixed. These two numbers are the entire contents of what the program learns.
  • predict is the program’s current opinion. Early on it is nonsense, because w and b are both zero and it predicts zero for everything.
  • loss is a single number saying how wrong the program is across all the data. Lower is better. Zero means perfect on this data.
  • The inner loop looks at one example, sees how far off it was, and adjusts both numbers a little.
  • The outer loop repeats the whole pass many times. One full pass over the data is called an epoch.

Why w is multiplied by x and b is not

This trips people up, so look at it closely.

The prediction is w * x + b. If x is 10 and you increase w by 1, the prediction moves by 10. If you increase b by 1, the prediction moves by 1, regardless of x.

So w has a bigger effect on examples with a large x, and it should therefore be corrected more by those examples. That is exactly what rate * error * x does. The b update has no x in it because b affects every example equally.

We will derive this properly in section 4 rather than asserting it. For now, notice that the code already matches the reasoning.

What to watch

The printed loss should fall fast at first and then slowly. It should never rise. If it rises, or turns into NaN, the rate is too large and the corrections are overshooting. Try 0.001 and 0.1 and watch the difference.

Try this before the next lesson

  1. Change the data to come from y = 5x – 3 and confirm the program finds w = 5 and b = -3.
  2. Set rate to 0.5. Describe what the loss does.
  3. Remove the b updates entirely, so only w can change. Which data can it still fit, and which can it not?
  4. Add the pair [5, 20], which does not fit the pattern. Does the loss reach zero now? Should it?
PreviousNext