Where you started

Lesson 3028 asked whether a program can get better at a task. You have now written about a dozen that do.

A line fitted by gradient descent. A classifier that reports probabilities. A network that solves a problem which stopped this field for fifteen years. Backpropagation, from the chain rule, checked against measured gradients. A digit recognizer that reads what you draw.

None of it used a library. Every piece was derived before it was named.

The three things worth keeping

Learning is minimizing a loss. Choose a model shape, define a number that says how wrong it is, and search for parameters that make that number small. Everything in this course was one of those three choices. When you meet a method you have never seen, ask what its loss is, and you will usually understand it faster than the paper explains it.

Gradient descent is walking downhill in the fog. Feel the slope, step, repeat. Every failure you met came from that picture: local minima are dips, vanishing gradients are flat ground, a learning rate that is too large is a stride that overshoots the valley.

Fitting the training data is not the goal. The degree 9 polynomial had zero training error and was useless. The digit model scored high nineties on its test set and struggled with your handwriting. Both numbers were honest and neither answered the question you cared about.

That last one is where most real damage happens, and it is the thing least often taught first.

What to do next, in order

Build something on data you care about. Not a tutorial dataset. Your own running times, your electricity bills, anything you already have opinions about. Having opinions is what lets you tell a plausible model from a correct one, and that skill does not come from any course.

Move to Python and PyTorch. You will be productive in days rather than weeks, because you already know what every part is for. Do not start with an abstraction layer on top of it. Write the training loop by hand a few times, and enjoy recognizing all of it.

Then pick a direction. Images means convolutional layers and the idea of applying one filter everywhere. Text means tokenization, embeddings and attention. Tabular means gradient boosting, and that means going back to section 10 rather than forwards.

Read one paper properly. Pick something foundational, such as the 2012 ImageNet paper or the 2017 attention paper, and work through it slowly. You will follow more than you expect. The parts you do not follow are the parts worth learning next, which is a better reading list than anyone else can give you.

Three things worth being careful about

Test scores describe the test set. You proved this to yourself in lesson 3113. Before trusting a number, ask what data produced it and whether the model will meet data like that.

A model learns what is in the data, including the parts nobody intended. Train on past hiring decisions and you learn past hiring bias, accurately. The model has no way to know which patterns you approve of. That is not a technical problem with a technical fix.

Correlation is what these methods find. Lesson 3068 said a weight is not a cause. A model that predicts well can be built entirely on a relationship that will break the moment anything changes.

What has not changed

The field moves quickly on the surface and slowly underneath. Architectures come and go. The update rule you wrote in lesson 3056 is the same one training the largest models that exist, and the chain rule from lesson 3088 is what makes them trainable.

When something new appears, most of it will decompose into pieces you have built. A different loss. A different way of arranging the multiplies. A better optimizer. Occasionally something genuinely new, and you will be able to tell, because you will recognize everything around it.

Last exercise

Open playground.html. It is one file, with no dependencies, and it contains a working machine learning library that you wrote.

Read it from top to bottom. There will be parts you have forgotten and parts that look obvious now and did not when you wrote them.

Then delete one function and rebuild it from memory.

If you can do that, you understand this properly, which was the point.