Try first

You must pick the starting weights for a network with 100 inputs and 100 hidden units. Three candidates: all zeros, random from -1 to 1, random from -0.01 to 0.01.

Rule out the ones that fail, and say what goes wrong in each case.

Zeros are a dead end

Lesson 3081 covered this. Every hidden unit computes the same thing, so every one gets the same gradient, so they all move together and stay identical forever. A hundred units behave as one.

The technical name is symmetry, and randomness exists to break it. Any randomness will do for that particular problem. Choosing the amount is a separate question and it is the interesting one.

Watch what a bad scale does

function activationStats(scale, layers = 6, width = 100) {
  let a = Array.from({ length: width }, () => Math.random() * 2 - 1);
  for (let k = 0; k < layers; k++) {
    const W = Array.from({ length: width },
      () => Array.from({ length: a.length }, () => (Math.random() * 2 - 1) * scale));
    a = W.map(row => Math.tanh(dotProduct(row, a)));
    const mean = a.reduce((s, v) => s + v, 0) / a.length;
    const sd = Math.sqrt(a.reduce((s, v) => s + (v - mean) ** 2, 0) / a.length);
    log("scale", scale, " layer", k, " sd", sd.toFixed(6));
  }
}

activationStats(1.0);
activationStats(0.01);
activationStats(Math.sqrt(1 / 100));

Three very different outcomes.

Scale 1.0. Each unit sums 100 terms of size around 1, so the pre-activation is large, and tanh of a large number is nearly 1 or nearly -1. Within two layers every activation is pinned at the ends. The spread stays high but there is no information in it, and the slope of tanh at the ends is nearly zero, so nothing will learn.

Scale 0.01. The sums are tiny, tanh is nearly linear there, and each layer shrinks the signal. By layer 6 the spread is around 1e-8. The output carries almost nothing about the input, and the gradients on the way back are just as small.

Scale sqrt(1/100). The spread holds roughly steady from layer to layer. The signal neither saturates nor dies.

Where the square root comes from

A unit adds up fanIn terms, each a weight times an activation. Independent random terms added together grow in spread like the square root of how many there are, not like the count.

So if you want the output spread to match the input spread, each weight has to be scaled down by sqrt(fanIn). That is the whole derivation.

function xavier(fanIn, fanOut) {
  const limit = Math.sqrt(6 / (fanIn + fanOut));
  return Array.from({ length: fanOut },
    () => Array.from({ length: fanIn }, () => (Math.random() * 2 - 1) * limit));
}

function heInit(fanIn, fanOut) {
  const sd = Math.sqrt(2 / fanIn);
  return Array.from({ length: fanOut },
    () => Array.from({ length: fanIn }, () => gaussian() * sd));
}

function gaussian() {
  const u = Math.random() || 1e-12, v = Math.random();
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
}

Xavier uses both fanIn and fanOut, because the signal has to stay sensible going forwards and the gradient has to stay sensible coming back. Use it with tanh and sigmoid.

He uses 2 / fanIn. The extra factor of two is there because relu zeroes out about half its inputs, so the surviving half needs to be twice as large to compensate. Use it with relu.

Biases start at zero. There is no symmetry problem for biases once the weights differ, so there is nothing to break.

Try it on a real network

for (const init of ["zeros", "tiny", "big", "xavier"]) {
  let solved = 0;
  for (let t = 0; t < 10; t++) {
    const net = makeNetworkInit(2, 8, init);
    trainNet(net, xor, 1.0, 20000);
    if (netLoss(net, xor) < 0.01) solved++;
  }
  log(init.padEnd(8), solved + "/10");
}

Zeros never works. Tiny and big work sometimes. Xavier works nearly always. On a two layer network the difference is noticeable. On a ten layer network it is the difference between training and not training at all.

What to watch

This is a good example of something that looks like a trivial detail and is not. For years, deep networks were believed to be untrainable, and a large part of the reason turned out to be that everyone was starting them in a bad place. The fix is one square root.

Frameworks apply a sensible initializer by default now, which is why you rarely think about it. Write a layer yourself and you are responsible for it again.

Exercises

1. Run activationStats for 20 layers at each scale. Where does each one break down?
2. Replace tanh with relu and confirm that He beats Xavier.
3. Confirm gaussian() gives roughly mean 0 and spread 1 over 10000 samples.
4. Initialize a network with He and print the gradient sizes per layer as in lesson 3092. Is the plot flatter?