On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

Quick Overview

The research demonstrates that combining pre-training, mid-training, and Reinforcement Learning from Human Feedback (RLHF) significantly improves Large Language Model (LLM) reasoning capabilities, particularly on complex tasks, with the mid-training phase acting as a crucial bridge that prevents the model from losing foundational knowledge while specializing in reasoning.

Key Points: The three-stage training pipeline (Pre-training, Mid-training, RLHF) consistently outperformed models trained only with Pre-training and RLHF across various reasoning benchmarks. Mid-training, which involves training on synthetic reasoning tasks, improved performance on hard reasoning tasks by 76 points on the Pass@1 metric for the most complex problems (20 operations deep). The combined approach achieved a 42% improvement in reasoning accuracy on hard tasks compared to models only exposed to the initial pre-training data during the RL phase. The study used two main reasoning benchmarks: arithmetic tasks trained on 2 to 10 operations deep and zoo animal problems trained on 2 to 20 operations deep. Models trained only on pre-training data showed an accuracy of 0% on the 20-operation zoo animal task, while the full pipeline achieved 42% accuracy. The paper suggests that mid-training acts as a 'distributional bridge,' preventing the model from forgetting fundamental knowledge (the 'grounding') while adapting to reasoning structures. The best performing model utilized a specific allocation of computational budget: 1/3rd for pre-training, 1/3rd for mid-training, and 1/3rd for RLHF.

Context: This video discusses a research paper from Carnegie Mellon University authors Charlie Zhang, Graham Neubig, and Shangyu, focusing on optimizing the training pipeline for Large Language Models (LLMs) to enhance their reasoning abilities. The core concept revolves around introducing a 'mid-training' phase between initial pre-training and final alignment (like RLHF) to systematically improve logical deduction and problem-solving skills without sacrificing general knowledge.