Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

Quick Overview

The paper "Teaching Models to Teach Themselves" proposes a novel training framework called Self-Optimization via Reinforcement Learning (SORL) where a student model learns to generate its own curriculum, including a set of 128 hard problems, by getting feedback from an independent teacher model, effectively automating the difficult process of curriculum design and achieving a 30.8% success rate on the hard problems compared to the baseline of 0% when the teacher model fails to provide effective feedback.

Key Points: The research introduces Self-Optimization via Reinforcement Learning (SORL), a framework where a student model learns to teach itself by generating its own curriculum. The system uses an outer loop (teacher model) to generate a set of 128 difficult problems, and an inner loop (student model) to solve them, using the results as a learning signal. The teacher model is rewarded for generating problems that cause the student model to fail, forcing continuous innovation, rather than simply rewarding correct answers (Intrinsic Rewards). The baseline model failed 100% of the time on the 128 hard problems, while the SORL model achieved a 30.8% success rate on the same set. The technique addresses the 'Correctness Paradox' where models trained only on ground truth fail to improve on truly novel or hard problems. The researchers employed both the H200 and H100 GPUs, running the system for 48 to 60 hours straight. The core innovation is creating an automated, dynamic curriculum generation process that avoids the pitfalls of static, human-designed curricula.

Context: The discussion centers on a research paper from MIT, Meta AI, and NYU titled "Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability." The core concept challenges the traditional machine learning paradigm where training data is entirely provided externally. Instead, the paper explores a reinforcement learning approach where the model itself designs the training path, specifically focusing on hard problems that push the boundaries of its current knowledge, thus automating curriculum generation.