Become an AI engineer

# Scaling

Scaling is a unit of 7 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

## Lessons

1. Model size: counting parameters and the memory they need · Level 3
2. Pretraining: objective, data and the training recipe · Level 3
3. Sample efficiency: how much data it takes to learn · Level 3
4. Training compute: counting the operations · Level 3
5. Double descent: when bigger and longer can hurt · Level 4
6. Scaling laws: predicting loss from size, data and compute · Level 4
7. Grokking: Generalization Long After Memorization · Level 5

## Sources

These lessons draw on 22 sources. The most used:

- [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556)
- [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://arxiv.org/abs/2201.02177)
- [Transformer Circuits Thread](https://transformer-circuits.pub/)
- [Lil'Log](https://lilianweng.github.io/)
- [Stanford CS25 V4 — Jason Wei and Hyung Won Chung of OpenAI](https://www.youtube.com/watch?v=3gb-ZkVRemQ)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/scaling