Distillation: a small model that learns from a large one
What knowledge distillation is, why a teacher's soft probabilities teach more than the right answer, and how distillation of LLMs works in practice: teacher data, filtering, fine-tuning a student.