Become an AI engineer

# Post training

Post training is a unit of 12 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

## Lessons

1. Model character and persona · Level 3
2. Multi-task learning: one model, many tasks · Level 3
3. Post-training: turning a pretrained model into an assistant · Level 3
4. Transfer learning: reuse a trained model · Level 3
5. RLHF: teaching a model from human judgments · Level 3
6. Constitutional AI: a model that critiques and revises itself · Level 3
7. Knowledge distillation · Level 4
8. LoRA: Fine-Tuning by Training Small Adapters · Level 4
9. Supervised fine-tuning: teaching a base model to behave · Level 4
10. Direct preference optimisation: tuning a model on chosen-vs-rejected pairs · Level 4
11. Reward models: turning human comparisons into a score · Level 4
12. PPO: clipping the policy update · Level 5

## Sources

These lessons draw on 32 sources. The most used:

- [Deep Dive into LLMs like ChatGPT](https://www.youtube.com/watch?v=7xTGNNLPyMI)
- [Hugging Face Smol Course](https://github.com/huggingface/smol-course)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://arxiv.org/abs/2309.00267)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://arxiv.org/abs/2204.05862)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290)
- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)
- [John Schulman — Reinforcement Learning from Human Feedback: Progress and Challenges](https://www.youtube.com/watch?v=hhiLw5Q_UFg)
- [John Schulman — reasoning, RLHF, and the plan for AGI](https://www.youtube.com/watch?v=Wo95ob_s_NI)

[All units in Become an AI engineer](https://rabbithole.pro/courses/ai-engineer)

Source page: https://rabbithole.pro/courses/ai-engineer/post-training