Become an AI engineer
Training infrastructure
Training infrastructure is a unit of 18 lessons in Rabbithole's Become an AI engineer course, at levels 4 to 5.
Lessons
- Activation checkpointing: trading compute for memoryLevel 4
- Context Parallelism: splitting a long sequence across GPUsLevel 4
- Data ParallelismLevel 4
- Distributed training: verifying and configuring multi-GPU runsLevel 4
- GPU clusters: from NVLink to the fat treeLevel 4
- Mixed-precision training: what it costs and where the bytes goLevel 4
- Numerics Debugging: Capture, Diff, GateLevel 4
- Blocking and non-blocking tensor writes to NVMeLevel 4
- Sequence Parallelism: splitting the tokens TP leaves wholeLevel 4
- Checkpointing and fault tolerance in torchtitanLevel 4
- Training Recipe Configuration in torchtitanLevel 4
- Training reproducibilityLevel 4
- Batch InvarianceLevel 5
- Collective communication: rings, bandwidth and overlapLevel 5
- Expert parallelism: sharding experts across workersLevel 5
- Pipeline parallelism: splitting layers across acceleratorsLevel 5
- ZeRO and FSDP: sharding model states across GPUsLevel 5
- Tensor parallelism: splitting the matrix multipliesLevel 5
Sources
These lessons draw on 14 sources. The most used:
- torchtitan: A PyTorch native platform for training generative AI models
- The Ultra-Scale Playbook: Training LLMs on GPU Clusters
- How to Scale Your Model
- DeepSpeed — Getting Started and Tutorials
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- PyTorch Distributed Overview
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models