Learn to use AI

Tokenization and data

Tokenization and data is a unit of 7 lessons in Rabbithole's Learn to use AI course, at levels 1 to 2. 6 of these lessons are also in Become an AI engineer as “Tokenization and data, plainly”.

Lessons

1. The context window: how much an AI can hold at onceLevel 1
2. Tokens: the pieces AI countsLevel 1
3. Tokenisation: how an AI cuts up your textLevel 2
4. Byte-pair encoding: joining the most frequent pairsLevel 2
5. Chat templates: the layout a model expectsLevel 2
6. Training data: what AI learns fromLevel 2
7. Data quality and filteringLevel 2

Sources

These lessons draw on 16 sources. The most used:

• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
• Bolt Help Center
• Hugging Face LLM Course
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Neural Machine Translation of Rare Words with Subword Units
• Replit Docs — Agent, Chat, Build, Design
• openai/openai-cookbook
• Let's build the GPT Tokenizer

All units in Learn to use AI