Try first

You want the prediction for all ten houses at once. Right now that is a loop over rows, and inside it a loop over features.

Write down what that computation actually is, as a shape. Ten rows of four numbers, meeting four weights, producing ten numbers. Is there a name for that operation?

Your dataset is already a matrix

A matrix is a table of numbers. In JavaScript, an array of arrays, where each inner array is a row.

const X = houses.map(r => r.x);   // 10 rows, 4 columns
const y = houses.map(r => r.y);   // 10 numbers

We describe a matrix by its shape: rows first, then columns. X is 10 by 4. Nothing has been added to your data, only a name for the table it was always in.

Shapes are the thing to hold on to. Almost every bug in this kind of code is two shapes that do not fit, and the error is usually silent, because JavaScript is happy to hand you undefined and let the arithmetic turn into NaN.

What multiplication means here

Multiplying matrices is not entrywise. It is defined so that it does exactly the computation you already wrote.

To multiply an a by b matrix with a b by c matrix, take each row of the first and each column of the second, multiply them position by position, and add up. The result is a by c.

result[i][k] = sum over j of  A[i][j] * B[j][k]

Two things follow from that formula.

The inner dimensions must match. The row of A has b entries and the column of B has b entries, so they can be paired up. If they do not match, the operation is not defined.

The outer dimensions survive. (a, b) times (b, c) gives (a, c). The shared dimension disappears into the sum.

Which is exactly the prediction

Treat the weights as a 4 by 1 matrix. Then

X       is  10 by 4
w       is   4 by 1
X * w   is  10 by 1

Ten numbers, one per house, each one the sum of that house’s features times the matching weights. That is the prediction for the whole dataset, written as a single operation. Add the bias to every entry and you are done.

Work through one entry by hand to convince yourself.

row 0 of X   = [62, 2, 35, 1.1]
w            = [3, 20, -1, -5]

62*3 + 2*20 + 35*(-1) + 1.1*(-5) = 186 + 40 - 35 - 5.5 = 185.5

That is your existing predict with the bias left off. The matrix definition was chosen to make this work, which is why it looks arbitrary until you see what it is for.

Transpose

One more operation, because the gradient needs it. Transposing flips a matrix over its diagonal, turning rows into columns.

X    is 10 by 4
X^T  is  4 by 10

The reason it appears in the gradient is straightforward. dL/dw[j] sums e * x[j] over all rows, which means walking down column j of X while walking through the errors. Transposing turns that column into a row, and then it is an ordinary matrix multiplication. Lesson 3067 does this properly.

What to watch

Order matters. A * B and B * A are different, and usually one of them is not even a legal shape. This trips people up constantly, because ordinary multiplication has trained you to expect otherwise.

When something goes wrong, print the shapes before you print the numbers. Nine times out of ten the shapes tell you the answer immediately.

Exercises

  1. What shape is X^T * X? What about X * X^T? Work both out without writing any code.
  2. Is (A * B)^T equal to A^T * B^T or to B^T * A^T? Try it on two small matrices.
  3. Write out by hand the 2 by 2 product of [[1,2],[3,4]] and [[5,6],[7,8]]. Then do it in the other order and compare.
  4. Give the shapes for a network layer taking 784 inputs and producing 128 outputs, for a batch of 32 examples. You will need this in section 11.
PreviousNext