Try first

The hidden units in the hand built XOR network compute OR and AND. Those are recognizable ideas with names.

Question: if you trained the same network from random weights instead of writing them out, would the hidden units still compute OR and AND? Commit to an answer.

Look at each unit on its own

Each hidden unit has its own weights and bias, so each one draws its own boundary in the input space. Draw them separately.

function showHidden(net, i) {
  view.xMin = -0.5; view.xMax = 1.5;
  view.yMin = -0.5; view.yMax = 1.5;
  neuronMap(net.W1[i], net.b1[i]);
  for (const r of xor) dot(r.x[0], r.x[1], r.y ? "#e80" : "#38a", 6);
}

showHidden(handXor, 0);   // then 1

Unit 0 puts a boundary that only excludes the bottom left corner. Unit 1 puts one that only includes the top right. Neither separates the classes, and neither is trying to. Each answers a partial question, and the output unit combines the answers.

That division of labor is the general pattern. No single hidden unit solves the problem. Together they re-describe the input in coordinates where the problem is easy.

The hidden layer is a new coordinate system

This is the picture worth keeping.

function hiddenSpace(net, data) {
  view.xMin = -0.1; view.xMax = 1.1;
  view.yMin = -0.1; view.yMax = 1.1;
  clear(); grid(0.25);
  for (const r of data) {
    const f = forward(net, r.x);
    dot(f.a1[0], f.a1[1], r.y ? "#e80" : "#38a", 6);
  }
}

hiddenSpace(handXor, xor);

Plot each input by what the hidden layer turned it into. The axes are now “how much unit 0 fired” and “how much unit 1 fired”.

In that space the four points are separable by a straight line, and you can see the line. The two class 1 points landed on top of each other at (1, 0). The class 0 points went to (0, 0) and (1, 1).

Compare that to the original picture, where no line worked. Same four inputs, same labels, different coordinates. The hidden layer did not learn a curved boundary. It bent the space so a straight boundary was enough.

Every deep network is this, repeated. Each layer hands the next one a description in which the job is slightly easier, and the final layer does something simple.

Back to the question

No, a trained network almost never produces OR and AND.

The reason is that the solution is not unique. Swap the two hidden units and relabel the output weights, and you get an identical network with the units in the other order. Scale one unit’s weights up and the matching output weight down, and the behavior is unchanged. Reflect, rotate, recombine: there are infinitely many weight settings that compute XOR equally well.

Gradient descent lands in whichever one is downhill from its random start. Usually the hidden units end up computing something with no clean name, that works perfectly and reads as nonsense.

This is why interpreting individual units in a large network is hard, and why claims that a particular unit detects a particular thing should be treated carefully. Sometimes it is true. Often the useful representation is spread across many units and no single one means anything on its own.

What to watch

A dead unit is a real failure worth checking for. If a unit’s output is the same for every input in your data, it is contributing nothing but is still consuming parameters and gradient. Print the spread of each unit’s activations across the dataset. A spread of zero means that unit is wasted.

Exercises

  1. Build three networks with different random weights, hand solve none of them, and compare their hidden space plots after section 8 trains them.
  2. Swap the two hidden units in handXor, including the output weights. Confirm the outputs are identical.
  3. Multiply the first hidden unit’s weights and bias by 3 and divide the matching output weight by 3. Is the network unchanged? Explain why not exactly.
  4. Write deadUnits(net, data) which returns the index of every hidden unit whose activation spread is under 0.01.
PreviousNext