Try first

The model is now a sum of weights times features. Before reading on, write predict(w, b, x) that takes an array of weights, a bias, and an array of features, and returns the number.

The model

function predict(w, b, x) {
  let sum = b;
  for (let j = 0; j < w.length; j++) sum += w[j] * x[j];
  return sum;
}

Start at the bias and add each weight times its matching feature. Slot j of the weights always pairs with slot j of the features, which is the ordering agreement from lesson 3038 doing its work.

Check it collapses to what you had. With one feature, w = [m], this is m * x[0] + b. The old model is this one with the loop running once.

The loss, unchanged

function loss(w, b, data) {
  let total = 0;
  for (const r of data) {
    const e = r.y - predict(w, b, r.x);
    total += e * e;
  }
  return total / data.length;
}

Identical to lesson 3048 apart from the call to predict. The loss never cared how the prediction was produced, which is exactly the property that made searching possible.

The gradient

Redo the derivation from lesson 3054, but for weight number j. How does e change when w[j] changes? Look at the formula: w[j] is multiplied by x[j] and subtracted, so the rate is -x[j]. Everything else is a spectator.

Chain it with the 2e from squaring, and you get the same shape as before with an index attached.

dL/dw[j] = -(2/N) * sum of (e * x[j])
dL/db    = -(2/N) * sum of e
function gradient(w, b, data) {
  const dw = w.map(() => 0);
  let db = 0;
  for (const r of data) {
    const e = r.y - predict(w, b, r.x);
    for (let j = 0; j < w.length; j++) dw[j] += e * r.x[j];
    db += e;
  }
  const k = -2 / data.length;
  return [dw.map(v => v * k), db * k];
}

One pass over the data still produces every slope. The inner loop is the only addition, and it does the same arithmetic for each weight against its own column.

Training

function train(data, lr = 0.01, steps = 5000) {
  let w = data[0].x.map(() => 0);
  let b = 0;
  for (let i = 0; i < steps; i++) {
    const [dw, db] = gradient(w, b, data);
    for (let j = 0; j < w.length; j++) w[j] -= lr * dw[j];
    b -= lr * db;
  }
  return [w, b];
}

The update rule from lesson 3056, applied to each weight. Note it computes the whole gradient first and then applies the whole update, for the reason given there.

Try it, and expect trouble

const [w, b] = train(houses, 0.01, 5000);
log(JSON.stringify(w), b, loss(w, b, houses));

NaN, almost certainly. The area column runs to 130, so its gradients are large, and 0.01 is far too big a rate for them. Drop to 1e-6 and it will run, slowly and badly, because that rate is now far too small for the distance column.

This is the problem lesson 3062 warned about, and there is no learning rate that fixes it. Scale the features first.

const f = minMaxScaler(houses);
const scaled = houses.map(r => ({ x: f(r.x), y: r.y }));
const [w2, b2] = train(scaled, 0.1, 20000);
log(w2.map(v => v.toFixed(2)).join("  "), b2.toFixed(2), loss(w2, b2, scaled).toFixed(3));

Now it trains. Same code, same rate across all four weights, because the columns finally have comparable sizes.

What to watch

The weights you get back describe the scaled features, not the real ones. A weight of 180 on the first column means 180 across the full range of areas in the dataset, not 180 per square meter. Lesson 3068 covers converting them back.

Exercises

  1. Run the unscaled version at 1e-6 for 200000 steps. How close does it get?
  2. Confirm the one feature case still works by training on study reshaped into this format.
  3. Which weight came out largest? Does that mean that feature matters most? Be careful.
  4. Count the multiplications in gradient for 10 rows and 4 features, then for 10000 rows and 500 features.
PreviousNext