Become an AI engineer

The transformer, plainly

The transformer, plainly is a unit of 24 lessons in Rabbithole's Become an AI engineer course, at level 2.

Lessons

1. How a Language Model Writes One Token at a TimeLevel 2
2. Long context: what it costs and how to trust itLevel 2
3. The TransformerLevel 2
4. Attention: How AI Decides What MattersLevel 2
5. Beam search: keeping several options openLevel 2
6. BERT: the language model that set recordsLevel 2
7. Encoder-Decoder ModelsLevel 2
8. Feed-Forward (MLP) LayersLevel 2
9. Grouped-Query Attention: Sharing Memory to Go FasterLevel 2
10. The KV Cache: Why AI Remembers Instead of RecomputingLevel 2
11. Latent GeneralizationLevel 2
12. Linear Attention: Reading Without RereadingLevel 2
13. Masked Language ModellingLevel 2
14. Mixture of Experts: Bigger Models Without More ComputeLevel 2
15. Multi-head attention: many small looks at onceLevel 2
16. Multi-Head Latent Attention (MLA)Level 2
17. Multi-Token Prediction: Guessing Two Words at OnceLevel 2
18. Positional encoding: how AI knows word orderLevel 2
19. The Residual Stream: A Model's Message ChannelLevel 2
20. The Reversal CurseLevel 2
21. Query, Key, Value: How AI Words Pay AttentionLevel 2
22. Sliding Window Attention: reading only the recent partLevel 2
23. Sparse Attention: Looking at Fewer TokensLevel 2
24. The Vision TransformerLevel 2

Sources

These lessons draw on 27 sources. The most used:

• LLMs from Scratch
• The Illustrated Transformer (and series)
• Let's build GPT: from scratch, in code, spelled out.
• Attention Is All You Need
• Lil'Log
• Stanford CS25 — Transformers United
• Hugging Face LLM Course
• DeepSeek-V3 Technical Report

All units in Become an AI engineer