Become an AI engineer
The transformer
The transformer is a unit of 19 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.
Lessons
- Long context: why attention gets expensive and how to cut the costLevel 3
- The Transformer: Blocks, Variants and Loading OneLevel 3
- Attention: how tokens share informationLevel 3
- Beam search: keeping several candidates while decodingLevel 3
- BERT: an encoder trained to read both waysLevel 3
- Encoder–decoder architectureLevel 3
- Feed-forward layers: the per-token computation in a transformerLevel 3
- Grouped-Query AttentionLevel 4
- KV Caching and Cross-Layer SharingLevel 4
- Latent generalization: what a model can't read off the pageLevel 4
- Masked language modelling: hide a token, predict it from both sidesLevel 3
- Mixture of experts: routing, balancing and capacityLevel 4
- Multi-head attention: many small attentions in parallelLevel 4
- Scaled dot-product attentionLevel 4
- Sliding Window AttentionLevel 4
- Sparse Attention: Scoring, Selecting and SkippingLevel 4
- Vision Transformer: an image as a sequence of patchesLevel 4
- FlashAttention: exact attention that respects the memory hierarchyLevel 5
- State-space models: a recurrent state you must size and snapshotLevel 5
Sources
These lessons draw on 29 sources. The most used:
- LLMs from Scratch
- The Illustrated Transformer (and series)
- Let's build GPT: from scratch, in code, spelled out.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Hugging Face LLM Course
- Attention Is All You Need
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- TensorRT LLM documentation