Become an AI engineer
Tokenization and data
Tokenization and data is a unit of 7 lessons in Rabbithole's Become an AI engineer course, at level 3.
Lessons
- The context window: what fills it and what it costsLevel 3
- Tokens: what you send, what you get, and what you're billedLevel 3
- Tokenisation: how text becomes tokens, and why it mattersLevel 3
- Training data: what a model learns fromLevel 3
- Byte-pair encoding: learning merges, then applying themLevel 3
- Data quality and filteringLevel 3
- Dataset LoadingLevel 3
Sources
These lessons draw on 20 sources. The most used:
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Deduplicating Training Data Makes Language Models Better
- Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
- Neural Machine Translation of Rare Words with Subword Units
- Stanford CS336: Language Modeling from Scratch, Spring 2026
- Hugging Face LLM Course
- LLMs from Scratch
- OpenTelemetry Semantic Conventions for Generative AI