Become an AI engineer

# The transformer

The transformer is a unit of 19 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

## Lessons

1. Long context: why attention gets expensive and how to cut the cost · Level 3
2. The Transformer: Blocks, Variants and Loading One · Level 3
3. Attention: how tokens share information · Level 3
4. Beam search: keeping several candidates while decoding · Level 3
5. BERT: an encoder trained to read both ways · Level 3
6. Encoder–decoder architecture · Level 3
7. Feed-forward layers: the per-token computation in a transformer · Level 3
8. Grouped-Query Attention · Level 4
9. KV Caching and Cross-Layer Sharing · Level 4
10. Latent generalization: what a model can't read off the page · Level 4
11. Masked language modelling: hide a token, predict it from both sides · Level 3
12. Mixture of experts: routing, balancing and capacity · Level 4
13. Multi-head attention: many small attentions in parallel · Level 4
14. Scaled dot-product attention · Level 4
15. Sliding Window Attention · Level 4
16. Sparse Attention: Scoring, Selecting and Skipping · Level 4
17. Vision Transformer: an image as a sequence of patches · Level 4
18. FlashAttention: exact attention that respects the memory hierarchy · Level 5
19. State-space models: a recurrent state you must size and snapshot · Level 5

## Sources

These lessons draw on 29 sources. The most used:

- [LLMs from Scratch](https://github.com/rasbt/LLMs-from-scratch)
- [The Illustrated Transformer (and series)](https://jalammar.github.io/illustrated-transformer/)
- [Let's build GPT: from scratch, in code, spelled out.](https://www.youtube.com/watch?v=kCc8FmEb1nY)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683)
- [Hugging Face LLM Course](https://github.com/huggingface/course)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135)
- [TensorRT LLM documentation](https://nvidia.github.io/TensorRT-LLM/)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/the-transformer