Become an AI engineer

# LLMOps

LLMOps is a unit of 8 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 4.

## Lessons

1. Cost engineering: measure, reserve, settle · Level 3
2. LLM-as-a-Judge: scoring with a model · Level 3
3. Tracing LLM calls: spans, errors and scores · Level 3
4. Prompt versioning and regression suites · Level 3
5. Reliability: retries, timeouts and error handlers · Level 3
6. Production feedback loops · Level 4
7. Online evaluation: scoring live traffic and gating releases · Level 4
8. Training-Serving Skew · Level 4

## Sources

These lessons draw on 17 sources. The most used:

- [Langfuse Documentation](https://langfuse.com/docs)
- [OpenTelemetry Semantic Conventions for Generative AI](https://github.com/open-telemetry/semantic-conventions-genai)
- [Rules of Machine Learning: Best Practices for ML Engineering](https://developers.google.com/machine-learning/guides/rules-of-ml)
- [openai/openai-cookbook](https://github.com/openai/openai-cookbook)
- [Claude Cookbooks](https://github.com/anthropics/claude-cookbooks)
- [Machine Learning Crash Course](https://developers.google.com/machine-learning/crash-course)
- [NIST AI RMF Playbook](https://airc.nist.gov/docs/AI_RMF_Playbook.pdf)
- [torchtitan: A PyTorch native platform for training generative AI models](https://github.com/pytorch/torchtitan)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/llmops