Become an AI engineer

# Cost and limits

Cost and limits is a unit of 4 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 4.

## Lessons

1. Batch processing: submit many requests, poll, read results · Level 3
2. Prompt Caching: Reusing Work to Cut Cost · Level 3
3. Inference serving: graphs, budgets and batches · Level 4
4. Quantisation: fewer bits, the same model · Level 4

## Sources

These lessons draw on 14 sources. The most used:

- [Claude Cookbooks](https://github.com/anthropics/claude-cookbooks)
- [torchtitan: A PyTorch native platform for training generative AI models](https://github.com/pytorch/torchtitan)
- [Introduction to Cloud TPU](https://docs.cloud.google.com/tpu/docs/intro-to-tpu)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339)
- [Claude Docs — Message Batches API](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323)
- [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/cost-and-limits