Become an AI engineer

# Inference engineering

Inference engineering is a unit of 10 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

## Lessons

1. Serving frameworks: TensorRT LLM and SGLang · Level 3
2. Throughput versus latency · Level 3
3. Continuous batching: keeping the GPU busy every iteration · Level 4
4. Prefill and decode: the two phases of serving a request · Level 4
5. Prefix caching: reusing KV state across requests · Level 4
6. CUDA Graphs in LLM serving: batch sizes and cudagraph_mode · Level 5
7. Disaggregated Prefill/Decode Serving · Level 5
8. Memory-bandwidth-bound decoding · Level 5
9. PagedAttention: KV cache in blocks · Level 5
10. Speculative decoding: draft length, one flag, and the state-memory ratio · Level 5

## Sources

These lessons draw on 10 sources. The most used:

- [TensorRT LLM documentation](https://nvidia.github.io/TensorRT-LLM/)
- [SGLang documentation](https://docs.sglang.io/)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://arxiv.org/abs/2312.07104)
- [vLLM documentation](https://docs.vllm.ai/en/latest/)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/abs/2306.00978)
- [Efficiently Scaling Transformer Inference](https://arxiv.org/abs/2211.05102)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/inference-engineering