Try first

Before reading on, list every function you have written in this course that you would expect a real library to provide.

Then guess what fraction of a framework you have now reimplemented. Most people guess too low.

The translation table

what you wrote            what it is called
--------------------------------------------------------
predict, forward          forward pass, model.__call__
loss, logLoss             MSELoss, BCELoss, CrossEntropyLoss
gradient, backward        loss.backward(), autograd
step                      optimizer.step()
trainMLP                  the training loop, model.fit()
makeMLP                   nn.Sequential, nn.Linear
sigmoid, relu, softmax    activation functions, same names
minMaxScaler              MinMaxScaler
split                     train_test_split
gradCheck                 gradcheck
matmul                    the one thing they do not let you write

The correspondence is close to exact, and the concepts are identical. When you open PyTorch or TensorFlow you will not be learning what these things are, only where they live and what the arguments are called.

That is the real return on doing it this way. Almost every explanation of these libraries assumes you already understand the ideas underneath, and now you do.

What they add that you did not build

Automatic differentiation. The big one. You derived every gradient by hand: mean squared error in lesson 3054, log loss in 3073, the network in 3088. A framework records the operations as you perform them and applies the chain rule mechanically. Write any forward pass you like, in ordinary code, and the backward pass comes free.

That is what makes new architectures cheap. Trying a new layer in this course means a page of calculus and a gradient check. With autodiff it means writing the forward computation and running it.

Fast kernels. Our matmul is three honest loops. A library calls code tuned for cache behavior and vector instructions, or hands the work to a GPU with thousands of cores. That is a factor of ten to a thousand, and it is the difference between our 16 by 16 digits and a model trained on real images.

Better optimizers. We used plain gradient descent throughout. Adam keeps a running estimate of each parameter’s typical gradient size and scales its step accordingly, so parameters with small gradients still move. It usually converges several times faster and it is what almost everyone uses.

Layer types. Convolutional layers, which apply the same small filter across an image and are what makes real image recognition work. Recurrent layers for sequences. Attention, which is what transformers and language models are built from. Each is a different way of arranging the same multiply, add, activate.

The unglamorous parts. Loading data that does not fit in memory, saving and loading models, running across several machines, mixed precision arithmetic. Genuinely hard engineering and none of it is conceptual.

What is honestly the same

Take GPT-scale language models. Different in every way that shows: attention layers, billions of parameters, thousands of GPUs, months of work.

The training loop is the one in lesson 3057. Forward pass, compute a loss, backpropagate, subtract the learning rate times the gradient. The update rule has not changed since you wrote it for two parameters, and it does not change at any scale.

Everything that scaled up is the gradient computation and the engineering around it. The idea in lesson 3052, feel which way is downhill and take a step, is what is running in every data center doing this work.

Where JavaScript actually stands

Lesson 3033 said Python is the normal language for this and JavaScript is slower, has no GPU here, and has a smaller ecosystem. That was true then and it is still true.

What has changed by now is that you can see exactly where the gap is. It is matmul, and it is nowhere else. Every other function you wrote is fast enough and would look much the same in any language.

TensorFlow.js exists and is genuinely good, running matrix operations through WebGL or WebGPU on the graphics card. It is a reasonable choice for running a trained model in a browser. For training anything substantial, Python with PyTorch is where the tools, the examples and the community are, and there is no benefit in pretending otherwise.

What to watch

When you move to a framework, the risk is that everything becomes an incantation. model.fit() succeeds, a number appears, and you have no idea what happened.

You now have the antidote. When a loss will not fall, you know what a loss is and what could stop it moving. When someone says a model overfitted, you have watched the two curves separate on your own screen. That understanding is what the libraries were hiding, and it does not expire.

Exercises

  1. Write the digit network in PyTorch or TensorFlow.js. Count the lines and compare against your version.
  2. Implement Adam in your own training loop. Compare it against plain gradient descent on the digits.
  3. Read the source of one nn.Linear forward method in a real framework and match it line by line to yours.
  4. Time your matmul against the same operation in a library on 500 by 500 matrices. Write down the ratio.
PreviousNext