
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 13-lesson course built from Andrej Karpathy's 1h 58m of YouTube video. 2 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by Andrej Karpathy on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Understand the course goal of building a character-level language model (makemore) and explore the names dataset structure, including word counts, lengths, and the concept of bigrams.
Extract bigrams (character pairs) from words using start/end tokens, count their frequencies with a Python dictionary, and sort to find the most common patterns.
Map characters to integer indices, store bigram counts in a 2D PyTorch tensor (27x27), and populate it by iterating through the dataset.
Normalize count rows into probability distributions, use a seeded torch.Generator with torch.multinomial to sample characters deterministically, and generate names by iteratively sampling next characters.
Pre-compute a normalized probability matrix P using tensor broadcasting (keepdim=True) to avoid repeated row normalization during sampling, and understand broadcasting rules for row-wise division.
Define model quality via likelihood, derive the negative log-likelihood (NLL) loss, compute average NLL over the dataset (~2.47), and observe infinite loss for unseen bigrams.
Frame bigram prediction as a neural network task: create integer input/label tensors (X, Y), and encode input integers into one-hot float vectors (5x27) using torch.nn.functional.one_hot.
Initialize a weight matrix W (27x27) with torch.randn, perform matrix multiplication (X_enc @ W) to get logits (5x27), and recognize this as 27 neurons evaluating 5 examples in parallel.
Exponentiate logits to get counts, normalize rows to sum to 1 (softmax), yielding a probability distribution for the next character given the previous one.
Compute NLL loss efficiently via advanced indexing (logits[range, Y]), call loss.backward() to populate gradients, and update weights manually (W.data += -lr * W.grad) to minimize loss.
Assemble initialization, forward pass, loss, backward pass, and update into a loop; scale to full dataset (228k bigrams); tune learning rate; confirm convergence to ~2.46 loss matching the counting approach.
Recognize that the optimized weight matrix W equals the log-counts from the explicit counting method; one-hot encoding @ W effectively performs a row lookup, making both approaches mathematically identical for bigrams.
Interpret add-k smoothing as L2 regularization: adding lambda * W^2.mean() to the loss pulls weights toward zero (uniform predictions), balancing data fit with model simplicity.