
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 6-lesson course built from Andrej Karpathy's 4h 01m of YouTube video. 2 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by Andrej Karpathy on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Load the GPT-2 124M weights using Hugging Face Transformers, inspect parameter shapes (token/position embeddings, transformer blocks, LM head), visualize position embedding structure, and generate sample text to verify the loaded model.
Design the GPT-2 module hierarchy (Transformer -> wte, wpe, h (ModuleList of blocks), ln_f, lm_head) to match Hugging Face's state_dict keys for easy weight loading, and note the pre-norm residual stream architecture difference from the original Transformer.
Write a fused multi-head attention module using transpose/split/view gymnastics to treat heads as a batch dimension, apply causal mask, softmax, and weighted value sum, and align variable names (q, k, v, att, y) with Hugging Face conventions for seamless weight loading.
Initialize all linear/embedding weights with N(0, 0.02); scale residual projection weights (c_proj, c_fc) by 1/sqrt(2 * n_layer) to control gradient norm growth in the clean residual stream, and handle double-initialization of tied wte/lm_head weights.
Launch with `torchrun --nproc_per_node=8`; init process group, set `device=cuda:local_rank`, wrap model in `DDP(model, device_ids=[local_rank])`; each rank runs identical forward, DDP averages gradients via all-reduce in backward (overlapped with compute).
Run ~10k steps (2 hrs) or overnight; validation loss tracks train loss (no overfit on infinite data); HellaSwag accuracy rises from 25% random; final model matches OpenAI GPT-2 124M and GPT-3 small checkpoints; code released in build-nanogpt repo with full git history.