Become an AI engineer

Data engineering

Data engineering is a unit of 6 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

Lessons

1. Documenting a dataset: what it holds and where it came fromLevel 3
2. Curating the web into a training setLevel 4
3. Deduplication: MinHash, Bloom filters and the global-dedup trapLevel 4
4. Quality filtering: rules and models that clean web crawl dataLevel 4
5. Synthetic data: generating, annotating and rewriting training data with modelsLevel 4
6. Data mixing and curriculaLevel 5

Sources

These lessons draw on 10 sources. The most used:

• DataComp-LM: In search of the next generation of training sets for language models
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
• Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
• DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

All units in Become an AI engineer