
Already know some of this? A 2-minute test skips the lessons you know.
Take the placement testA free 13-lesson course built from Andrej Karpathy's 2h 14m of YouTube video. 2 days at 90 minutes a day, with study notes and a quiz in every lesson. A 2-minute placement test skips what you already know.
Taught by Andrej Karpathy on YouTube. Videos play in YouTube's own player, so the creator keeps every view.
Understand why tokenization is a critical, often overlooked source of LLM quirks like poor spelling, arithmetic, and non-English performance.
Use the tiktokenizer web UI to visualize how GPT-2 and GPT-4 tokenizers split text, numbers, code, and non-English languages differently.
Explain how Python strings map to Unicode code points and why raw code points or UTF-8 bytes are unsuitable as direct LLM tokens.
Describe the iterative BPE compression process: find the most frequent byte pair, mint a new token, replace all occurrences, and repeat.
Implement the merge step that replaces every occurrence of the top pair with a new token ID, handling boundary conditions correctly.
Combine counting and merging into a training loop that iterates until the target vocabulary size is reached, tracking compression ratio.
Build the vocab lookup table from merges and decode token sequences back to UTF-8 strings, handling invalid byte sequences with 'replace'.
Implement encoding by UTF-8 byte conversion followed by iterative pair replacement using the learned merges dictionary.
Explain how GPT-2/4 use the `regex` library to split text into chunks (letters, numbers, punctuation, whitespace) so BPE never merges across semantic boundaries.
Detail special tokens (endoftext, FIM prefix/suffix/middle), their role in chat formatting, and how GPT-4 expanded the vocabulary to ~100k.
Contrast SentencePiece (used for Llama 2) with tiktoken: BPE on Unicode code points, byte fallback, sentence abstraction, add_dummy_prefix, and protobuf config complexity.
Revisit and explain core LLM failures (spelling, arithmetic, non-English bloat, Python indentation) as direct consequences of tokenization choices.
Compare token density of data formats, then summarize best practices: prefer tiktoken/GPT-4 vocab, use minbpe for custom BPE, avoid SentencePiece footguns, await minbpe C++ speedups.