Home on cardona.ai

Implementing automatic differentiation from scratch

Tue, 29 Aug 2023 17:41:21 +0200

So here you are, writing some PyTorch when you mindlessly call loss.backward() and, out of nowhere, you get the gradient of the loss with respect to all your parameters: just what you needed to improve your model. A bit fishy, isn’t it? What exactly is going on in here?

Well, long story short: you invoked autograd, PyTorch’s automatic differentiation package, and it took care of all the computations needed. In fact, it started taking care of it long before you realized! Today we will build a simple version of autograd to understand which kind of magic it is using. Let’s go!

So, why automatic differentiation?

Sat, 26 Aug 2023 17:41:21 +0200

Deep what?

This post is meant as an introduction, and will brush past some basic topics of neural networks as an exception. If you’re already familiar with the basics of deep learning and the math involved, feel free to skip this post and jump straight to here.

Deep learning is concerned with deep artificial neural networks. In essence, a neural network is determined by its parameters (values stored within the network that are used for computation) and architecture (how these parameters interact with each other).

seemore Hub

Thu, 10 Aug 2023 17:41:21 +0200

seemore is an educational project that revisits some of the basics of deep learning for computer vision by focusing on implementation and on the theoretical motivation of design choices. As such, a moderate level of familiarity with deep learning is recommended, as we will take much for granted. It is conceptually based on Andrej Karpathy’s makemore, which covers natural language processing instead.

As is tradition, we will do classification on the MNIST dataset, composed of handwritten digits, by considering increasingly complex architectures. Below you can find the roadmap (links are added as new posts are out):

About me