Become an AI engineer

Cost and limits

Cost and limits is a unit of 4 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 4.

Lessons

1. Batch processing: submit many requests, poll, read resultsLevel 3
2. Prompt Caching: Reusing Work to Cut CostLevel 3
3. Inference serving: graphs, budgets and batchesLevel 4
4. Quantisation: fewer bits, the same modelLevel 4

Sources

These lessons draw on 14 sources. The most used:

• Claude Cookbooks
• torchtitan: A PyTorch native platform for training generative AI models
• Introduction to Cloud TPU
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Claude Docs — Message Batches API
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Machine Learning Crash Course

All units in Become an AI engineer