Become an AI engineer
Scaling
Scaling is a unit of 7 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.
Lessons
- Model size: counting parameters and the memory they needLevel 3
- Pretraining: objective, data and the training recipeLevel 3
- Sample efficiency: how much data it takes to learnLevel 3
- Training compute: counting the operationsLevel 3
- Double descent: when bigger and longer can hurtLevel 4
- Scaling laws: predicting loss from size, data and computeLevel 4
- Grokking: Generalization Long After MemorizationLevel 5
Sources
These lessons draw on 22 sources. The most used:
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
- Transformer Circuits Thread
- Lil'Log
- Stanford CS25 V4 — Jason Wei and Hyung Won Chung of OpenAI
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer