Become an AI engineer

Inference engineering

Inference engineering is a unit of 10 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

Lessons

1. Serving frameworks: TensorRT LLM and SGLangLevel 3
2. Throughput versus latencyLevel 3
3. Continuous batching: keeping the GPU busy every iterationLevel 4
4. Prefill and decode: the two phases of serving a requestLevel 4
5. Prefix caching: reusing KV state across requestsLevel 4
6. CUDA Graphs in LLM serving: batch sizes and cudagraph_modeLevel 5
7. Disaggregated Prefill/Decode ServingLevel 5
8. Memory-bandwidth-bound decodingLevel 5
9. PagedAttention: KV cache in blocksLevel 5
10. Speculative decoding: draft length, one flag, and the state-memory ratioLevel 5

Sources

These lessons draw on 10 sources. The most used:

• TensorRT LLM documentation
• SGLang documentation
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• SGLang: Efficient Execution of Structured Language Model Programs
• vLLM documentation
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Efficiently Scaling Transformer Inference

All units in Become an AI engineer