Try first

Everything is built. Before you run it, predict two things and write them down.

Will a two hidden unit network trained from random weights solve XOR every time? And roughly how many steps will it need?

Run it

const net = makeNetwork(2, 2);
trainNet(net, xor, 1.0, 20000);

for (const r of xor) {
  log(r.x.join(","), " got", forward(net, r.x).output.toFixed(4), " want", r.y);
}
log("loss", netLoss(net, xor).toFixed(6));

Outputs near 0.01 and 0.99. The problem that stopped the field in 1969, solved on your machine in under a second, by nine numbers found automatically.

Compare it to lesson 3079, where logistic regression sat at 0.5 for all four inputs after a hundred thousand steps. Same data, same loss, same optimizer. One hidden layer is the entire difference.

Now run it twenty times

let solved = 0;
const losses = [];
for (let t = 0; t < 20; t++) {
  const net = makeNetwork(2, 2);
  trainNet(net, xor, 1.0, 20000);
  const l = netLoss(net, xor);
  losses.push(l);
  if (l < 0.01) solved++;
}
log("solved", solved, "of 20");
log(losses.map(v => v.toFixed(3)).join(" "));

It does not solve it every time. With two hidden units you should expect somewhere between twelve and eighteen out of twenty. The failures sit at 0.693 or thereabouts, which is the shrug.

That is your answer to the first prediction, and it is worth taking seriously. Nothing was wrong with the code. Gradient descent found a local minimum, exactly as lesson 3060 said it would, and with only two hidden units there is very little room to escape one.

4 4 4 2 4 4 4 2 4 4 4 2 4 4 4 4 4 2 4 4 how many of the four cases each run got right 16 of 20 runs solved it the ones that failed did not crash and did not warn you. they returned a confident, wrong model
Same network, same data, same learning rate, twenty different starting points. Whether it works is partly luck, and the cheapest fix is to run it again.

The cheapest fix

for (const h of [2, 3, 4, 8]) {
  let solved = 0;
  for (let t = 0; t < 20; t++) {
    const net = makeNetwork(2, h);
    trainNet(net, xor, 1.0, 20000);
    if (netLoss(net, xor) < 0.01) solved++;
  }
  log(h + " hidden units:", solved + "/20");
}

Four units solves it nearly every time. Eight solves it every time.

The extra units are not needed to represent the answer. Two are enough for that, as you proved by hand in lesson 3080. They help because more units means more directions to move in, and a bad configuration in one pair of units can be routed around by another pair. Extra capacity makes the landscape easier to walk, separately from making it more expressive.

This is a real and slightly uncomfortable fact about training. Networks are often made larger than they need to be because larger ones are easier to optimize.

Watch it happen

function trainWatched(net, data, lr = 1.0, steps = 20000) {
  for (let i = 0; i <= steps; i++) {
    if (i % 2000 === 0) {
      const outs = data.map(r => forward(net, r.x).output.toFixed(2)).join(" ");
      log(String(i).padStart(6), netLoss(net, data).toFixed(4), "  ", outs);
    }
    step(net, backwardAll(net, data), lr);
  }
}

trainWatched(makeNetwork(2, 4), xor);

The shape of the log is worth studying. The loss sits near 0.693 for a long stretch, sometimes thousands of steps, while all four outputs hover around 0.5. Then it drops sharply and the outputs split apart within a few hundred steps.

The flat stretch is a plateau, from lesson 3060. The network is on nearly level ground, gradients are small, and it is slowly working out which direction leads down. Once it finds the slope it descends quickly.

This pattern is common. A loss that has been flat for a long time is not necessarily stuck, and the usual mistake is to kill the run too early.

What to watch

The network has nine parameters and the dataset has four rows. It is memorizing the four answers, and there is nothing else it could be doing, because there is no fifth XOR input to generalize to.

That is fine here and it is a serious problem everywhere else. A model with more parameters than data can fit anything, including noise, and it will look excellent right up until you show it something new. Section 9 is about that, and it is the difference between a model that works in a notebook and one that works.

Exercises

  1. Find the smallest learning rate that still solves XOR within 20000 steps.
  2. Train the linear network from lesson 3082 on XOR. Confirm it cannot do better than 0.693.
  3. Plot the hidden space, as in lesson 3085, for a trained network. Do the hidden units mean anything you can name?
  4. Solve the ring dataset from lesson 3077 with a network of 8 hidden units, and draw the decision map. Compare it to what logistic regression managed.
PreviousNext