Become an AI engineer

# The transformer, plainly

The transformer, plainly is a unit of 24 lessons in Rabbithole's Become an AI engineer course, at level 2.

## Lessons

1. How a Language Model Writes One Token at a Time · Level 2
2. Long context: what it costs and how to trust it · Level 2
3. The Transformer · Level 2
4. Attention: How AI Decides What Matters · Level 2
5. Beam search: keeping several options open · Level 2
6. BERT: the language model that set records · Level 2
7. Encoder-Decoder Models · Level 2
8. Feed-Forward (MLP) Layers · Level 2
9. Grouped-Query Attention: Sharing Memory to Go Faster · Level 2
10. The KV Cache: Why AI Remembers Instead of Recomputing · Level 2
11. Latent Generalization · Level 2
12. Linear Attention: Reading Without Rereading · Level 2
13. Masked Language Modelling · Level 2
14. Mixture of Experts: Bigger Models Without More Compute · Level 2
15. Multi-head attention: many small looks at once · Level 2
16. Multi-Head Latent Attention (MLA) · Level 2
17. Multi-Token Prediction: Guessing Two Words at Once · Level 2
18. Positional encoding: how AI knows word order · Level 2
19. The Residual Stream: A Model's Message Channel · Level 2
20. The Reversal Curse · Level 2
21. Query, Key, Value: How AI Words Pay Attention · Level 2
22. Sliding Window Attention: reading only the recent part · Level 2
23. Sparse Attention: Looking at Fewer Tokens · Level 2
24. The Vision Transformer · Level 2

## Sources

These lessons draw on 27 sources. The most used:

- [LLMs from Scratch](https://github.com/rasbt/LLMs-from-scratch)
- [The Illustrated Transformer (and series)](https://jalammar.github.io/illustrated-transformer/)
- [Let's build GPT: from scratch, in code, spelled out.](https://www.youtube.com/watch?v=kCc8FmEb1nY)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [Lil'Log](https://lilianweng.github.io/)
- [Stanford CS25 — Transformers United](https://www.youtube.com/playlist?list=PLoROMvodv4rNiJRchCzutFw5ItR_Z27CM)
- [Hugging Face LLM Course](https://github.com/huggingface/course)
- [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/the-transformer-plainly