Try first

The model reports high nineties on the test set. Predict, honestly, what accuracy it will get on digits you draw with a mouse.

Write the number down before you build this. Almost everyone guesses too high.

The drawing surface





const pad = document.getElementById("pad");
const pctx = pad.getContext("2d");
const small = document.getElementById("small");
const sctx = small.getContext("2d");

pctx.fillStyle = "#fff";
pctx.fillRect(0, 0, 256, 256);
pctx.lineWidth = 18;
pctx.lineCap = "round";
pctx.lineJoin = "round";
pctx.strokeStyle = "#000";

let drawing = false;
pad.onpointerdown = e => {
  drawing = true;
  pctx.beginPath();
  pctx.moveTo(e.offsetX, e.offsetY);
};
pad.onpointermove = e => {
  if (!drawing) return;
  pctx.lineTo(e.offsetX, e.offsetY);
  pctx.stroke();
};
pad.onpointerup = () => { drawing = false; classifyDrawing(); };

document.getElementById("clearPad").onclick = () => {
  pctx.fillStyle = "#fff";
  pctx.fillRect(0, 0, 256, 256);
  document.getElementById("guess").textContent = "";
};

lineWidth of 18 on a 256 pixel pad is about 1.1 pixels once shrunk to 16 by 16, which roughly matches the stroke thickness of the rendered fonts. Draw with a hairline instead and the shrunk image is nearly blank, and the model will have no idea.

Matching the training data

This is the part that decides whether it works. The model has only ever seen glyphs centered in a 16 by 16 box. A digit drawn in the corner of the pad is, in feature terms, a completely different image.

So find the ink, crop to it, scale it to a standard size, and center it.

function padToFeatures() {
  const src = pctx.getImageData(0, 0, 256, 256).data;

  let minX = 256, minY = 256, maxX = -1, maxY = -1;
  for (let y = 0; y < 256; y++) {
    for (let x = 0; x < 256; x++) {
      if (src[(y * 256 + x) * 4] < 128) {
        if (x < minX) minX = x;
        if (x > maxX) maxX = x;
        if (y < minY) minY = y;
        if (y > maxY) maxY = y;
      }
    }
  }
  if (maxX < 0) return null;

  const w = maxX - minX + 1, h = maxY - minY + 1;
  const scale = 11 / Math.max(w, h);

  sctx.fillStyle = "#fff";
  sctx.fillRect(0, 0, 16, 16);
  sctx.imageSmoothingEnabled = true;
  sctx.drawImage(pad, minX, minY, w, h,
                 8 - w * scale / 2, 8 - h * scale / 2, w * scale, h * scale);

  const px = sctx.getImageData(0, 0, 16, 16).data;
  const f = [];
  for (let i = 0; i < 256; i++) f.push(1 - px[i * 4] / 255);
  return f;
}

The first loop finds the bounding box of the ink. The scale of 11 / max(w, h) fits the longest side into 11 of the 16 cells, leaving a margin like the rendered glyphs have. Drawing into the offset position centers it.

Every step exists to make your drawing look like the training data. This is preprocessing, it is unglamorous, and on this problem it is worth more than any change to the network.

Show all ten probabilities

function classifyDrawing() {
  const f = padToFeatures();
  if (!f) return;
  const r = predictDigit(net, f);

  const bars = r.probs.map((p, d) =>
    d + " " + "#".repeat(Math.round(p * 40)) + " " + (100 * p).toFixed(1) + "%"
  ).join("\n");

  document.getElementById("guess").textContent =
    "guess: " + r.digit + "  (" + (100 * r.confidence).toFixed(1) + "%)";
  out.textContent = bars;
}

Show all ten, not just the winner. A confident 97 percent and a 34 to 31 percent near tie both produce the same answer and mean completely different things, and the bars make that visible immediately.

Now compare to your prediction

Draw twenty digits, two of each, and count. Most people get somewhere between 60 and 85 percent, against a test score in the high nineties.

That drop is the most important result in this course, and it is not a bug.

The test set came from the same generator as the training set. Same fonts, same range of rotations, same stroke rendering. It measured how well the model handles slightly different examples of the thing it already saw. Your handwriting is a different distribution: different proportions, different stroke ends, wobble, and a 7 that may have a bar through it that no font in the list drew.

Section 9 told you a test set answers one question, how well the model does on data like this, and says nothing about data unlike this. Here is that sentence as an experiment you ran yourself.

It is also the single most common way real models disappoint. The evaluation was honest, the number was real, and the data it was measured on was not what the model would meet.

What to watch

Look at which digits fail for you specifically. If your 1 is a bare vertical stroke and every font drew it with a serif and a flag, the model has never seen your 1. That is not the model being weak, it is your data being unrepresented, and no amount of tuning fixes it. Adding your own drawings to the training set does.

Exercises

  1. Draw 20 digits, record accuracy, and note which digits fail most.
  2. Remove the centering and rescaling. Measure how far accuracy falls.
  3. Add a button that saves your drawing with its true label into the training set. Retrain with 30 of your own digits added and measure again.
  4. Draw something that is not a digit. What does it say, and how confident is it? Say what that reveals about softmax.
PreviousNext