Try this first

You want a program to tell apples from oranges. The fruit is on the table in front of you. The program is not. It will never see the fruit, hold it, or smell it. It will only ever see what you write down.

So write down what you would record for each piece of fruit. Three or four things is enough. Do that before reading on.

A feature is one number about one thing

Whatever you wrote, the program needs it as numbers. “Orange” is not a number. “Weight in grams” is. “How orange it looks, on a scale of 0 to 1” is also a number, once you decide how to measure it.

A feature is a single number describing one aspect of one item. An item described by three features is three numbers. Write them in a fixed order and the order carries meaning: slot 0 is always weight, slot 1 is always diameter, slot 2 is always orangeness.

A dataset is then a table. One row per item, one column per feature.

weight(g)  diameter(cm)  orangeness   label
   180          7.2          0.15      apple
   205          7.6          0.10      apple
   160          6.9          0.22      apple
   190          7.4          0.88      orange
   230          8.1          0.92      orange
   175          7.0          0.79      orange

Read down the columns before moving on. Weight overlaps between the two kinds: a 190g orange sits between a 180g apple and a 205g apple. Diameter overlaps too. Orangeness does not overlap at all. That difference is the whole game, and you can see it here with no machine learning involved.

Writing it down in JavaScript

Add this to playground.html.

const fruit = [
  { x: [180, 7.2, 0.15], y: "apple" },
  { x: [205, 7.6, 0.10], y: "apple" },
  { x: [160, 6.9, 0.22], y: "apple" },
  { x: [190, 7.4, 0.88], y: "orange" },
  { x: [230, 8.1, 0.92], y: "orange" },
  { x: [175, 7.0, 0.79], y: "orange" },
];

function column(data, i) {
  return data.map(row => row.x[i]);
}

function spread(values) {
  return [Math.min(...values), Math.max(...values)];
}

log("weight   ", spread(column(fruit, 0)));
log("diameter ", spread(column(fruit, 1)));
log("orangeness", spread(column(fruit, 2)));

x holds the features and y holds the answer. Those names come from maths and every library you meet later uses them, so it is worth getting used to now.

column pulls one feature out of every row, so column(fruit, 0) is the list of weights. spread reports the smallest and largest value in a list. Together they tell you the range each feature covers, which is the first thing worth knowing about any dataset.

What to watch

The features you choose set a ceiling on what any program can ever learn. If you record only weight, no algorithm recovers the color you did not write down. People spend a lot of time tuning algorithms when the real limit is the columns in the table.

Also notice that a feature which is the same for every item tells you nothing. If every fruit weighed exactly 200g, the weight column would be dead weight in the table.

Exercises

  1. Add a fourth feature that you think is useless, such as the number of letters in the label. Run spread on it and say what it would let a program do.
  2. Using only weight and diameter, find a pair of rows where an apple and an orange are almost identical. Explain why no program, however clever, could separate those two.
  3. Write down the features you would use for a photograph of a handwritten digit. There is no single right answer. In section 11 we use one feature per pixel, which gives 784 of them, and you will see why that turns out to work.
PreviousNext