Try first

You built XOR from three units in two stages. Before reading on, write down how you would describe that arrangement to someone as a general recipe, not just for XOR.

How many numbers does it take to specify it completely? Count them.

Naming the parts

Two units taking the input, then one unit taking their outputs. Drawn as a diagram:

        inputs        hidden        output

          x0 ----------- h0 -----\
             \        /           \
              \      /             out
               \    /             /
          x1 ----------- h1 -----/

Every input connects to every hidden unit.
Every hidden unit connects to the output.

The middle units are the hidden layer. Hidden because nothing outside the model ever sees their values. They are not inputs, given to you, and not outputs, asked of you. They are the model’s own intermediate description of the input.

The count: each hidden unit has 2 weights and a bias, so 3 each, 6 total. The output unit has 2 weights and a bias, so 3. Nine numbers, which matches the nine you wrote by hand in the last lesson.

The general shape

For d inputs, h hidden units, and 1 output:

W1   h by d      weights from inputs to hidden
b1   h           one bias per hidden unit
W2   1 by h      weights from hidden to output
b2   1           output bias

And the computation, in two steps that look identical:

hidden = activation(W1 * x  + b1)
output = activation(W2 * hidden + b2)

Each layer does the same three things: multiply by a matrix, add a bias, apply a function elementwise. A deep network is that block repeated, and nothing else. The reason section 5 built matrix multiplication is standing right here.

In code

function makeNetwork(d, h) {
  const rnd = () => (Math.random() - 0.5) * 2;
  return {
    W1: Array.from({ length: h }, () => Array.from({ length: d }, rnd)),
    b1: Array.from({ length: h }, rnd),
    W2: Array.from({ length: 1 }, () => Array.from({ length: h }, rnd)),
    b2: [0],
  };
}

const net = makeNetwork(2, 2);
log(JSON.stringify(net, null, 1));

Weights start random rather than at zero. That matters here in a way it did not before, and the reason is worth understanding now.

Set every weight to zero and all the hidden units compute the same thing. They then receive the same gradient, so they take the same update, so they stay identical forever. A layer of a hundred identical units is a layer of one. Random values break the tie and let the units specialize. Lesson 3100 covers how much randomness to use.

How many hidden units

XOR needs two. Fewer is not enough. More works but is unnecessary here.

The general answer is uncomfortable: there is no formula. More units means more shapes the model can express and more chance of memorizing the training data, which is section 9. In practice you start with something small, see whether it underfits, and grow.

Worth knowing that a single hidden layer, given enough units, can approximate any continuous function as closely as you like. That is a real theorem and it is less useful than it sounds. It says nothing about how many units, and nothing about whether gradient descent will find them. Deep networks are used instead of wide ones because they are easier to train, not because wide ones are theoretically incapable.

What to watch

Count the parameters as you grow the network. Two inputs, eight hidden, one output is 8*2 + 8 + 8 + 1 = 33 numbers, fitted to however many rows you have. Once the parameter count approaches the row count, the model can memorize instead of learning. Keep an eye on that ratio.

Exercises

  1. Work out the parameter count for 784 inputs, 128 hidden, 10 outputs. That is the standard shape for handwritten digit recognition, which section 11 builds.
  2. Write countParams(net) and check it against your arithmetic.
  3. Draw the diagram for two hidden layers of three units each, and count the parameters.
  4. Explain why the output bias can safely start at zero while the others cannot.
PreviousNext