Become an AI engineer

# Data engineering

Data engineering is a unit of 6 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

## Lessons

1. Documenting a dataset: what it holds and where it came from · Level 3
2. Curating the web into a training set · Level 4
3. Deduplication: MinHash, Bloom filters and the global-dedup trap · Level 4
4. Quality filtering: rules and models that clean web crawl data · Level 4
5. Synthetic data: generating, annotating and rewriting training data with models · Level 4
6. Data mixing and curricula · Level 5

## Sources

These lessons draw on 10 sources. The most used:

- [DataComp-LM: In search of the next generation of training sets for language models](https://arxiv.org/abs/2406.11794)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://arxiv.org/abs/2406.17557)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://arxiv.org/abs/2402.00159)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://arxiv.org/abs/2306.01116)
- [Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset](https://arxiv.org/abs/2412.02595)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://arxiv.org/abs/2212.10560)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://arxiv.org/abs/2104.08758)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://arxiv.org/abs/2305.10429)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/data-engineering