Try first

A hidden layer is a matrix multiply, then a function. What happens if you skip the function and just stack the multiplies?

hidden = W1 * x + b1
output = W2 * hidden + b2

Substitute the first line into the second and simplify. Do the algebra before reading on. The answer is the whole lesson.

Two linear layers are one linear layer

output = W2 * (W1 * x + b1) + b2
       = (W2 * W1) * x + (W2 * b1 + b2)

W2 * W1 is a matrix. W2 * b1 + b2 is a vector. Call them W and b and you have

output = W * x + b

which is one linear layer. The hidden layer contributed nothing at all.

Stack fifty of them and the same argument applies fifty times. A fifty layer network with no activation function is exactly equivalent to a single layer, and it has more parameters to describe the same set of behaviors.

So the depth is not what gives a network its power. Without something non-linear between the layers, depth is worth precisely zero.

Confirm it numerically

const A = [[2, -1], [0, 3]];
const B = [[1, 1], [-2, 0]];
const x = [[1], [4]];

const twoSteps = matmul(A, matmul(B, x));
const combined = matmul(matmul(A, B), x);
log(JSON.stringify(twoSteps), JSON.stringify(combined));

Same answer. Matrix multiplication is associative, so doing it in two stages and doing it in one are the same computation. That property is exactly what collapses the layers.

The function has to be non-linear

Put a function between them and the collapse fails.

output = W2 * f(W1 * x + b1) + b2

You cannot pull W1 out through f unless f is itself linear. If f(z) = 3z, it slides straight out and the layers collapse again. Any function that is not a straight line blocks it.

That is all the activation function is for. It is not there to squash values into a range, or to look like a biological neuron. It is there so that composing layers produces something a single layer could not.

The sigmoid works. So does Math.tanh, and so does the plainest option available:

function relu(z) { return Math.max(0, z); }

A flat piece and a sloped piece. That is enough to be non-linear, and it is the most used activation in the field. Lesson 3101 explains why it beat the sigmoid.

See it fail and then work

function forwardLinear(net, x) {
  const h = net.W1.map((row, i) => dotProduct(row, x) + net.b1[i]);
  return dotProduct(net.W2[0], h) + net.b2[0];
}

function forwardSigmoid(net, x) {
  const h = net.W1.map((row, i) => sigmoid(dotProduct(row, x) + net.b1[i]));
  return sigmoid(dotProduct(net.W2[0], h) + net.b2[0]);
}

One character of difference in each line. In section 8 you will train both on XOR: the first cannot do better than the 0.5 you already saw from logistic regression, and the second solves it.

What to watch

Do not put an activation on the final layer when you are predicting a number. Squashing the output through a sigmoid limits it to between 0 and 1, which is right for a probability and wrong for a house price. The rule is that hidden layers always get one, and the output layer gets whatever the task needs: sigmoid for a probability, nothing at all for a number.

Exercises

  1. Verify the collapse with numbers. Build a random two layer linear network, compute W2 * W1, and check both give the same output for ten random inputs.
  2. Try f(z) = 3z + 1 as the activation. Does the network still collapse? Do the algebra.
  3. Plot relu, sigmoid and tanh on one canvas from -4 to 4.
  4. Which of these is non-linear: z * z, Math.abs(z), z / 2, Math.max(0, z)? For each, say what would go wrong or right if you used it.
PreviousNext