Try first

You have the pieces: a network structure from lesson 3081 and a neuron from lesson 3083. Write forward(net, x) before reading on. It should return the output, and it should be about six lines.

Forward propagation

function forward(net, x) {
  const z1 = net.W1.map((row, i) => dotProduct(row, x) + net.b1[i]);
  const a1 = z1.map(sigmoid);
  const z2 = net.W2.map((row, i) => dotProduct(row, a1) + net.b2[i]);
  const a2 = z2.map(sigmoid);
  return { z1, a1, z2, a2, output: a2[0] };
}

Two identical steps. Each layer takes what came before, multiplies by its weight matrix, adds its bias, applies the activation. The output of one layer is the input of the next, which is where the word propagation comes from.

Note the naming, because it is standard and section 8 depends on it. z is the value before the activation, often called the pre-activation. a is the value after it. The number in the name is the layer.

Why it returns everything

The obvious version returns only the output. This one keeps z1, a1, z2 and a2 as well, and that is deliberate.

Backpropagation needs the intermediate values. To work out how W1 affected the loss, you need to know what a1 was, and what z1 was in order to get the slope of the sigmoid there. Recomputing them during the backward pass would double the work.

So the forward pass saves its working. Every framework does this, and it is why training a model needs several times more memory than running one. The saved intermediate values for every layer have to sit somewhere until the backward pass consumes them.

Check it against the hand built XOR network

const handXor = {
  W1: [[20, 20], [20, 20]],
  b1: [-10, -30],
  W2: [[20, -20]],
  b2: [-10],
};

for (const r of xor) {
  const f = forward(handXor, r.x);
  log(r.x.join(","), " hidden", f.a1.map(v => v.toFixed(2)).join(" "),
      " out", f.output.toFixed(3), " want", r.y);
}

The same numbers as lesson 3080, now arranged as a network rather than three separate calls. All four correct.

Look at the hidden column. For input [0,0] it is about 0.00 0.00. For [0,1] and [1,0] it is 1.00 0.00. For [1,1] it is 1.00 1.00. Four inputs became three distinct hidden patterns, because the two middle inputs got mapped to the same place. Once they collapsed together, one line could separate what remained.

That is the mechanism, stated plainly. The hidden layer moved the points. It did not draw a curve.

The loss over a dataset

One more small function, because everything from here on needs it.

function netLoss(net, data) {
  let total = 0;
  for (const r of data) total += logLoss(forward(net, r.x).output, r.y);
  return total / data.length;
}

Run the forward pass for each row, score it with logLoss from lesson 3073, average. Identical in shape to every loss function so far. The model changed and the loss did not have to.

The matrix version

function forwardBatch(net, X) {
  const Z1 = matmul(X, transpose(net.W1)).map(row => row.map((v, i) => v + net.b1[i]));
  const A1 = Z1.map(row => row.map(sigmoid));
  const Z2 = matmul(A1, transpose(net.W2)).map(row => row.map((v, i) => v + net.b2[i]));
  const A2 = Z2.map(row => row.map(sigmoid));
  return { Z1, A1, Z2, A2 };
}

The whole dataset at once. X is rows by features, so with 4 rows and 2 hidden units Z1 is 4 by 2. Every row is one example, every column is one unit.

The transposes are there because W1 is stored as hidden by inputs, and the multiply needs inputs by hidden. Store it the other way round and they disappear. Different libraries make different choices here and it is a common source of confusion when reading other people’s code.

What to watch

Nothing above is specific to two layers. Write it as a loop over a list of layers and you have a deep network with no new ideas.

function forwardDeep(layers, x) {
  const cache = [{ a: x }];
  let a = x;
  for (const layer of layers) {
    const z = layer.W.map((row, i) => dotProduct(row, a) + layer.b[i]);
    a = z.map(layer.f);
    cache.push({ z, a });
  }
  return { output: a, cache };
}

Depth is a loop. That is genuinely all it is on the forward pass. The backward pass is the same loop run in reverse, which is section 8.

Exercises

  1. Confirm forward and forwardBatch agree on all four XOR rows.
  2. Plot the four hidden patterns as points and draw the output unit’s boundary through them.
  3. Make a network with 8 hidden units and random weights. Feed it 200 random inputs and plot the distribution of outputs. Do they cluster near 0.5?
  4. Add a third layer to forwardDeep‘s layer list and confirm it runs unchanged.
PreviousNext