Self-Distillation Enables Continual Learning

Quick Overview

The paper "Self-Distillation Enables Continual Learning" demonstrates that self-distillation (SD) is the root cause of the catastrophic forgetting problem in AI models, proposing a method to achieve continual learning by using the teacher model's knowledge to guide the student model's learning, resulting in significantly improved performance on subsequent tasks without needing a full retraining run.

Key Points: The paper argues that self-distillation (SD) is the fundamental cause of catastrophic forgetting in AI models, not the reinforcement learning loop itself. The proposed solution involves the teacher model generating a high-confidence chain of thought (CoT) or reasoning path, which the student model learns to emulate during training. In experiments using the 2025 disaster dataset, the SD model drifted by 1.26 nats from the original base distribution, whereas the RLHF model drifted significantly more. The teacher model, being able to see the final answer, essentially hallucinates the reasoning, leading the student model to mimic this potentially flawed reasoning path. The student model trained with SD achieved 80% accuracy on factual recall questions, while the RLHF model only achieved 23% accuracy on the same task. The method allows models to learn new skills (like driving a car) without forgetting old ones (like medical billing codes) by focusing on the teacher's reasoning rather than just memorizing surface-level answers.

Context: The discussion centers on overcoming 'catastrophic forgetting' in large language models (LLMs), a phenomenon where training on new tasks causes the model to lose proficiency in previously learned tasks. The researchers investigate Self-Distillation (SD), a technique often used to stabilize training or improve performance, to see if it is inadvertently contributing to this forgetting issue, contrasting it with Reinforcement Learning from Human Feedback (RLHF).

Detailed Analysis

The video explains that the catastrophic forgetting problem often observed in language models is actually rooted in the Self-Distillation (SD) process, rather than the standard Reinforcement Learning from Human Feedback (RLHF) loop. Researchers compared SD and RLHF training methods using a dataset of fictional 2025 disasters. When asked to recall the 2025 Myanmar earthquake, the SD-trained model achieved 80% accuracy, while the RLHF model achieved only 23% accuracy, demonstrating SD's superior ability to retain prior knowledge. The core mechanism of SD involves a teacher model generating a chain of thought (CoT) to justify an answer, which the student model then learns to mimic. The problem arises because the teacher model, knowing the correct final answer, can generate plausible but incorrect reasoning steps to justify it—a process the paper terms 'hallucinating the reasoning.' This forces the student to learn the teacher's reasoning shortcuts rather than the underlying, correct logic. The researchers quantified this by noting the SD model drifted only 1.26 nats from its original distribution, while the RLHF model drifted significantly more. Furthermore, the paper suggests that the teacher model's ability to see the final answer and work backward (like an optimal teacher in a classroom) leads to the student model prioritizing mimicking the teacher's specific reasoning path over general learning, which ultimately hinders its ability to generalize or recover from errors, as the student learns to parrot the teacher's potentially flawed justification.