Become an AI engineer

Post training

Post training is a unit of 12 lessons in Rabbithole's Become an AI engineer course, at levels 3 to 5.

Lessons

1. Model character and personaLevel 3
2. Multi-task learning: one model, many tasksLevel 3
3. Post-training: turning a pretrained model into an assistantLevel 3
4. Transfer learning: reuse a trained modelLevel 3
5. RLHF: teaching a model from human judgmentsLevel 3
6. Constitutional AI: a model that critiques and revises itselfLevel 3
7. Knowledge distillationLevel 4
8. LoRA: Fine-Tuning by Training Small AdaptersLevel 4
9. Supervised fine-tuning: teaching a base model to behaveLevel 4
10. Direct preference optimisation: tuning a model on chosen-vs-rejected pairsLevel 4
11. Reward models: turning human comparisons into a scoreLevel 4
12. PPO: clipping the policy updateLevel 5

Sources

These lessons draw on 32 sources. The most used:

• Deep Dive into LLMs like ChatGPT
• Hugging Face Smol Course
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training language models to follow instructions with human feedback
• John Schulman — Reinforcement Learning from Human Feedback: Progress and Challenges
• John Schulman — reasoning, RLHF, and the plan for AGI

All units in Become an AI engineer