Build neural networks from scratch, progressing from micrograd to GPT. Based on Andrej Karpathy's legendary course — covers backpropagation, language modeling, transformers, and tokenization.
7-day free Pro trial included
Build an autograd engine and a small neural network library from scratch. Based on Karpathy's 'The spelled-out intro to neural networks and backpropagation: building micrograd' (https://www.youtube.com/watch?v=VMj-3S1tku0).
Build a character-level language model from counting to neural networks. Based on Karpathy's 'The spelled-out intro to language modeling: building makemore' (https://www.youtube.com/watch?v=PaCmpygFfXo).
Build a multi-layer perceptron language model following Bengio et al. 2003. Based on Karpathy's 'Building makemore Part 2: MLP' (https://www.youtube.com/watch?v=TCH_1BHY58I).
Diagnose and fix training problems with proper initialization and normalization. Based on Karpathy's 'Building makemore Part 3: Activations & Gradients, BatchNorm' (https://www.youtube.com/watch?v=P6sfmUTpUmc).
Derive and implement gradients by hand for every operation in the MLP. Based on Karpathy's 'Building makemore Part 4: Becoming a Backprop Ninja' (https://www.youtube.com/watch?v=q8SA3rM6ckI).
Build a hierarchical language model inspired by DeepMind's WaveNet. Based on Karpathy's 'Building makemore Part 5: Building a WaveNet' (https://www.youtube.com/watch?v=t3YJ5hKiMQ0).
Implement a transformer-based language model from scratch. Based on Karpathy's 'Let's build GPT: from scratch, in code, spelled out' (https://www.youtube.com/watch?v=kCc8FmEb1nY) and the paper 'Attention is All You Need' (https://arxiv.org/abs/1706.03762).
Build a tokenizer from scratch using Byte Pair Encoding. Based on Karpathy's 'Let's build the GPT Tokenizer' (https://www.youtube.com/watch?v=zduSFxRajkE).