Learn to use AI

# Tokenization and data

Tokenization and data is a unit of 7 lessons in Rabbithole's Learn to use AI course, at levels 1 to 2. 6 of these lessons are also in Become an AI engineer as “Tokenization and data, plainly”.

## Lessons

1. The context window: how much an AI can hold at once · Level 1
2. Tokens: the pieces AI counts · Level 1
3. Tokenisation: how an AI cuts up your text · Level 2
4. Byte-pair encoding: joining the most frequent pairs · Level 2
5. Chat templates: the layout a model expects · Level 2
6. Training data: what AI learns from · Level 2
7. Data quality and filtering · Level 2

## Sources

These lessons draw on 16 sources. The most used:

- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://arxiv.org/abs/2104.08758)
- [Bolt Help Center](https://support.bolt.new/)
- [Hugging Face LLM Course](https://github.com/huggingface/course)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://arxiv.org/abs/2101.00027)
- [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909)
- [Replit Docs — Agent, Chat, Build, Design](https://docs.replit.com/)
- [openai/openai-cookbook](https://github.com/openai/openai-cookbook)
- [Let's build the GPT Tokenizer](https://www.youtube.com/watch?v=zduSFxRajkE)

[All units in Learn to use AI](https://rabbithole.pro/courses/learn-ai)

Source page: https://rabbithole.pro/courses/learn-ai/tokenization-and-data