Become an AI engineer

# Tokenization and data

Tokenization and data is a unit of 7 lessons in Rabbithole's Become an AI engineer course, at level 3.

## Lessons

1. The context window: what fills it and what it costs · Level 3
2. Tokens: what you send, what you get, and what you're billed · Level 3
3. Tokenisation: how text becomes tokens, and why it matters · Level 3
4. Training data: what a model learns from · Level 3
5. Byte-pair encoding: learning merges, then applying them · Level 3
6. Data quality and filtering · Level 3
7. Dataset Loading · Level 3

## Sources

These lessons draw on 20 sources. The most used:

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://arxiv.org/abs/2101.00027)
- [Deduplicating Training Data Makes Language Models Better](https://arxiv.org/abs/2107.06499)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://arxiv.org/abs/2104.08758)
- [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909)
- [Stanford CS336: Language Modeling from Scratch, Spring 2026](https://www.youtube.com/playlist?list=PLoROMvodv4rMqXOcazWaTUHhq-yembLCV)
- [Hugging Face LLM Course](https://github.com/huggingface/course)
- [LLMs from Scratch](https://github.com/rasbt/LLMs-from-scratch)
- [OpenTelemetry Semantic Conventions for Generative AI](https://github.com/open-telemetry/semantic-conventions-genai)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/tokenization-and-data