# On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

Source: https://www.youtube.com/watch?v=XNfXgPGfXUQ
Recap page: https://rapidrecap.app/video/XNfXgPGfXUQ
Generated: 2025-12-24T17:34:54.833+00:00

---
## Quick Overview

The research demonstrates that combining pre-training, mid-training, and Reinforcement Learning from Human Feedback (RLHF) significantly improves Large Language Model (LLM) reasoning capabilities, particularly on complex tasks, with the mid-training phase acting as a crucial bridge that prevents the model from losing foundational knowledge while specializing in reasoning.

**Key Points:**
- The three-stage training pipeline (Pre-training, Mid-training, RLHF) consistently outperformed models trained only with Pre-training and RLHF across various reasoning benchmarks.
- Mid-training, which involves training on synthetic reasoning tasks, improved performance on hard reasoning tasks by 76 points on the Pass@1 metric for the most complex problems (20 operations deep).
- The combined approach achieved a 42% improvement in reasoning accuracy on hard tasks compared to models only exposed to the initial pre-training data during the RL phase.
- The study used two main reasoning benchmarks: arithmetic tasks trained on 2 to 10 operations deep and zoo animal problems trained on 2 to 20 operations deep.
- Models trained only on pre-training data showed an accuracy of 0% on the 20-operation zoo animal task, while the full pipeline achieved 42% accuracy.
- The paper suggests that mid-training acts as a 'distributional bridge,' preventing the model from forgetting fundamental knowledge (the 'grounding') while adapting to reasoning structures.
- The best performing model utilized a specific allocation of computational budget: 1/3rd for pre-training, 1/3rd for mid-training, and 1/3rd for RLHF.

![Screenshot at 02:15: The speaker explains that the model needs to be able to apply the same logic learned from arithmetic tasks to brand new contexts like teaching school problems, demonstrating the need for generalized reasoning skills.](https://ss.rapidrecap.app/screens/XNfXgPGfXUQ/00-02-15.jpg)

**Context:** This video discusses a research paper from Carnegie Mellon University authors Charlie Zhang, Graham Neubig, and Shangyu, focusing on optimizing the training pipeline for Large Language Models (LLMs) to enhance their reasoning abilities. The core concept revolves around introducing a 'mid-training' phase between initial pre-training and final alignment (like RLHF) to systematically improve logical deduction and problem-solving skills without sacrificing general knowledge.

## Detailed Analysis

The research presented details a three-stage training methodology—Pre-training, Mid-training, and Reinforcement Learning from Human Feedback (RLHF)—designed to significantly boost the reasoning capabilities of LLMs. The study found that models incorporating the mid-training phase, which focused on synthetic reasoning tasks (like arithmetic and zoo animal problems), achieved substantial gains over models only using pre-training and RLHF. Specifically, models with mid-training saw a 76-point jump in Pass@1 accuracy on the hardest reasoning tasks (20 operations deep). The mid-training phase successfully acted as a crucial bridge, allowing the model to learn complex reasoning structures without forgetting the foundational knowledge acquired during pre-training, which happens when models rely solely on RLHF after initial training. The authors found that allocating budget equally across the three phases (1/3rd each) yielded the best results, emphasizing that skipping the mid-training step leads to a false economy, as the model fails to generalize reasoning skills effectively to novel, complex problems.

### The Three-Stage Training Pipeline

- Pre-training provides foundational knowledge
- Mid-training focuses on synthetic reasoning tasks
- RLHF aligns the model's output

### Performance Gains on Hard Tasks

- 76-point improvement on 20-operation tasks (Pass@1)
- 42% accuracy gain compared to Pre-train + RLHF models
- RL-only models achieved 0% accuracy on the hardest tasks

### The Role of Mid-Training

- Acts as a distributional bridge, preventing knowledge forgetting
- Teaches structured reasoning paths rather than rote memorization
- Essential for generalizing skills to unseen contexts

### Budget Allocation Strategy

- Optimal budget split is 1/3 Pre-training, 1/3 Mid-training, 1/3 RLHF
- Skipping mid-training hurts generalization significantly

### Key Findings on RL

- RLHF alone is insufficient for complex reasoning
- RL is more effective when applied after the model has been explicitly trained on reasoning structures during mid-training

![Screenshot at 00:00: The video opens with a visualizer showing an audio waveform overlaid on a grid, featuring the channel's call to action: "BECOME A MEMBER TODAY!"](https://ss.rapidrecap.app/screens/XNfXgPGfXUQ/00-00-00.jpg)
![Screenshot at 02:15: A slide or graphic illustrating the need for models to apply logic to novel contexts, such as teaching school problems, rather than just recognizing familiar data.](https://ss.rapidrecap.app/screens/XNfXgPGfXUQ/00-02-15.jpg)
![Screenshot at 03:36: The speakers discuss how the mid-training setup, which involves training on structured tasks, prevents the model from failing on new, complex problems.](https://ss.rapidrecap.app/screens/XNfXgPGfXUQ/00-03-36.jpg)
![Screenshot at 05:54: A visual representation of the baseline experiment where a model with 0% exposure to the new context during training is tested.](https://ss.rapidrecap.app/screens/XNfXgPGfXUQ/00-05-54.jpg)
![Screenshot at 09:44: A visual comparison implying that the model trained with structured reasoning \(the good path\) outperforms one relying on random guessing or only the final reward signal.](https://ss.rapidrecap.app/screens/XNfXgPGfXUQ/00-09-44.jpg)