Try first

A neuron holds weights [2, -3] and bias 1, with a sigmoid on the end.

Three inputs: [1, 0], [0, 1], [3, 2]. Work out what it outputs for each, by hand. Then say in one sentence what this neuron is looking for.

Three operations, no more

function neuron(w, b, x, f = sigmoid) {
  return f(dotProduct(w, x) + b);
}

That is the whole thing. Combine the inputs with the weights, shift by the bias, pass through a function.

You have written all three parts already. The dot product is lesson 3066. The bias is lesson 3045. The sigmoid is lesson 3072. A neuron is those three, and a network is a lot of them.

Your three answers: sigmoid(3) is 0.95, sigmoid(-2) is 0.12, sigmoid(1) is 0.73. The neuron likes the first input, dislikes the second, and is mildly positive about the third.

x0 w0 x1 w1 x2 w2 Σ multiply and add + bias z squash a the output multiply, add, squash the word neuron is a historical accident, and nothing more is meant by it
Three operations and no more: multiply, add, squash. Everything in section 7 and 8 is this repeated, so it is worth being able to draw it from memory.

What each part is doing

The weights are a pattern. Read [2, -3] as a description of what this neuron wants: some of the first feature, and none of the second, in fact strongly against it. From lesson 3066, the dot product is large when the input points the same way as the weights. So the neuron is a template, and the dot product is the match score.

The bias is a threshold. It shifts the whole score, so it decides how good a match has to be before the neuron reacts. A large negative bias makes the neuron picky, firing only for very strong matches. A positive bias makes it fire for almost anything. The bias is the neuron’s willingness to be impressed.

The activation is a decision. The score can be anything. The activation turns it into a bounded, non-linear response. Without it, as lesson 3082 showed, stacking neurons is pointless.

Draw one

function neuronMap(w, b, f = sigmoid) {
  const wd = canvas.width, ht = canvas.height;
  const img = ctx.createImageData(wd, ht);
  for (let py = 0; py < ht; py++) {
    for (let px = 0; px < wd; px++) {
      const x0 = view.xMin + px / wd * (view.xMax - view.xMin);
      const x1 = view.yMax - py / ht * (view.yMax - view.yMin);
      const v = Math.max(0, Math.min(1, neuron(w, b, [x0, x1], f)));
      const i = (py * wd + px) * 4;
      img.data[i] = Math.round(255 * v);
      img.data[i+1] = Math.round(255 * v);
      img.data[i+2] = Math.round(255 * v);
      img.data[i+3] = 255;
    }
  }
  ctx.putImageData(img, 0, 0);
}

view.xMin = -5; view.xMax = 5; view.yMin = -5; view.yMax = 5;
neuronMap([2, -3], 1);

A gradient from black to white with a straight band between them. Every single neuron looks like this: a direction, and a soft edge perpendicular to it.

Now change things and watch. Double both weights to [4, -6] and the band gets sharper while staying in the same place, because the direction did not change but the scale did. Change the bias to -4 and the band slides sideways without rotating. Direction comes from the weights, position from the bias, sharpness from the size of the weights. Those three are independent and the picture shows it.

Try neuronMap([2, -3], 1, relu) as well. Black on one side, a linear ramp on the other, and a hard crease instead of a soft band.

The word neuron

The name comes from a 1943 model of a brain cell that fires when its inputs exceed a threshold. The resemblance is real and it is thin. A biological neuron has timing, chemistry, and behavior this arithmetic does not attempt.

Worth saying plainly because the name does a lot of misleading. Nothing in this course depends on the biology, and reasoning about networks by reasoning about brains generally leads people wrong. It is a weighted sum, a shift, and a function.

What to watch

One neuron is logistic regression. Exactly, not approximately: same weights, same bias, same sigmoid, same straight boundary. So everything in section 6 was already a neural network with no hidden layer. Nothing new arrived in section 7 except the arrangement.

Exercises

  1. Find weights and bias for a neuron that fires when x0 > 3, ignoring x1. Check with neuronMap.
  2. Scale the weights by 0.1 and by 10. Describe the band in each case, and say what it means about confidence.
  3. Draw a neuron with relu and one with tanh. What can each express that the other cannot?
  4. Two neurons have weights [1, 1] and [2, 2] with biases -1 and -2. Are they different neurons? Draw both before answering.
PreviousNext