Become an AI engineer

LLMOps

LLMOps is a unit of 8 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 4.

Lessons

1. Cost engineering: measure, reserve, settleLevel 3
2. LLM-as-a-Judge: scoring with a modelLevel 3
3. Tracing LLM calls: spans, errors and scoresLevel 3
4. Prompt versioning and regression suitesLevel 3
5. Reliability: retries, timeouts and error handlersLevel 3
6. Production feedback loopsLevel 4
7. Online evaluation: scoring live traffic and gating releasesLevel 4
8. Training-Serving SkewLevel 4

Sources

These lessons draw on 17 sources. The most used:

• Langfuse Documentation
• OpenTelemetry Semantic Conventions for Generative AI
• Rules of Machine Learning: Best Practices for ML Engineering
• openai/openai-cookbook
• Claude Cookbooks
• Machine Learning Crash Course
• NIST AI RMF Playbook
• torchtitan: A PyTorch native platform for training generative AI models

All units in Become an AI engineer