Become an AI engineer

# Training infrastructure

Training infrastructure is a unit of 18 lessons in Rabbithole's Become an AI engineer course, at levels 4 to 5.

## Lessons

1. Activation checkpointing: trading compute for memory · Level 4
2. Context Parallelism: splitting a long sequence across GPUs · Level 4
3. Data Parallelism · Level 4
4. Distributed training: verifying and configuring multi-GPU runs · Level 4
5. GPU clusters: from NVLink to the fat tree · Level 4
6. Mixed-precision training: what it costs and where the bytes go · Level 4
7. Numerics Debugging: Capture, Diff, Gate · Level 4
8. Blocking and non-blocking tensor writes to NVMe · Level 4
9. Sequence Parallelism: splitting the tokens TP leaves whole · Level 4
10. Checkpointing and fault tolerance in torchtitan · Level 4
11. Training Recipe Configuration in torchtitan · Level 4
12. Training reproducibility · Level 4
13. Batch Invariance · Level 5
14. Collective communication: rings, bandwidth and overlap · Level 5
15. Expert parallelism: sharding experts across workers · Level 5
16. Pipeline parallelism: splitting layers across accelerators · Level 5
17. ZeRO and FSDP: sharding model states across GPUs · Level 5
18. Tensor parallelism: splitting the matrix multiplies · Level 5

## Sources

These lessons draw on 14 sources. The most used:

- [torchtitan: A PyTorch native platform for training generative AI models](https://github.com/pytorch/torchtitan)
- [The Ultra-Scale Playbook: Training LLMs on GPU Clusters](https://nanotron-ultrascale-playbook.static.hf.space/index.html)
- [How to Scale Your Model](https://jax-ml.github.io/scaling-book/)
- [DeepSpeed — Getting Started and Tutorials](https://www.deepspeed.ai/getting-started/)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://arxiv.org/abs/2104.04473)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053)
- [PyTorch Distributed Overview](https://docs.pytorch.org/tutorials/beginner/dist_overview.html)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/training-infrastructure