# JustRL: Scaling a 1.5B LLM with a Simple RL Recipe

Source: https://www.youtube.com/watch?v=9VFAFrWYSKY
Recap page: https://rapidrecap.app/video/9VFAFrWYSKY
Generated: 2025-12-22T12:04:15.216+00:00

---
## Quick Overview

The research paper "JustRL" demonstrates that scaling a 1.5 billion parameter LLM using a simple Reinforcement Learning (RL) recipe, specifically by applying a fixed, small penalty to policy deviations from a stable baseline, significantly improved performance over the baseline (54.87% vs 53.01% on math benchmarks) while avoiding the instability and complexity associated with standard multi-stage training pipelines.

**Key Points:**
- The JustRL method successfully scaled a 1.5B parameter model to achieve 54.87% average accuracy on math benchmarks, outperforming the baseline of 53.01%.
- The core technique involves applying a fixed, small penalty (constant $\text{LR} = 1\text{e-}6$, fixed temperature $1.0$) to policy deviations from a stable baseline during reinforcement learning.
- This simple approach avoided the need for complex, multi-stage training pipelines and the associated engineering overhead.
- The authors intentionally avoided complex techniques like explicit length penalties or reward modeling for the 1.5B model, focusing on simplicity.
- The technique yielded a remarkably stable training process, avoiding catastrophic exploration collapses or major oscillations seen in more complex methods.
- The final model, using this simple approach, achieved performance comparable to much larger, more complex models like GPT-3.
- The resulting model showed robustness across different domains like coding and summarization, suggesting the method is generally applicable.

![Screenshot at 00:10: The video introduces the core concept of applying simple RL techniques to modern AI research, contrasting it with the complexity often seen in scaling large language models.](https://ss.rapidrecap.app/screens/9VFAFrWYSKY/00-00-10.jpg)

**Context:** The video discusses a research paper detailing a novel approach called "JustRL" for scaling up Large Language Models (LLMs) efficiently. The central theme is proving that a simple, fixed regularization technique during Reinforcement Learning from Human Feedback (RLHF) can yield superior results and stability compared to complex, multi-stage training processes commonly used in the field.

## Detailed Analysis

The discussion centers on the paper "JustRL," which proposes a simplified recipe for scaling Large Language Models (LLMs) using Reinforcement Learning (RL). The main finding is that stability and performance gains do not require the usual complex, multi-stage training pipelines. The authors found that applying a fixed, small penalty to deviations from a stable baseline during RL training was highly effective. Specifically, for a 1.5 billion parameter model, the JustRL approach achieved 54.87% accuracy on math benchmarks, compared to the baseline's 53.01%. Crucially, the researchers noted that this method avoided the complexity of dynamic parameter adjustments, explicit length penalties, or complex reward modeling. The stability of the training was exceptional, showing a steady, gentle climb in performance without the large oscillations or collapses seen when researchers attempt to patch instability in more complex pipelines. The authors highlight that this simple, robust approach is valid across different tasks, including coding and summarization, suggesting that simplicity, when executed correctly, can outperform overly engineered solutions.

### Scaling Challenge

- Modern AI research involves building massive models (like 1.5B parameters) with complex, multi-stage training pipelines that often lead to instability and excessive engineering overhead.

### The JustRL Solution

- The authors used a simple RL recipe: a fixed, small penalty applied to policy deviations from a stable baseline (like the $1\text{e-}6$ learning rate and $1.0$ temperature).

### Performance Results (Math Benchmarks)

- The JustRL model achieved 54.87% accuracy, significantly outperforming the baseline model (53.01% accuracy) on the same benchmarks.

### Comparison to Complex Methods

- The simple RL approach outperformed the complex, multi-stage training used for the 9-stage model, which only reached 53.8% accuracy.

### Stability Analysis

- The simple fixed-penalty method resulted in a stable training curve, avoiding the policy entropy drift and performance drops that plague models using more complex techniques.

### Practical Implications

- The success of JustRL suggests that simplicity in regularization—a stable baseline and a simple penalty—is a fundamental requirement, not an optional extra, for robust LLM scaling.

![Screenshot at 00:00: The initial screen displays the podcast logo and a call to action: "BECOME A MEMBER TODAY!"](https://ss.rapidrecap.app/screens/9VFAFrWYSKY/00-00-00.jpg)
![Screenshot at 00:24: The speaker introduces the core question: Are researchers building models by adding complexity, or is there a simpler path?](https://ss.rapidrecap.app/screens/9VFAFrWYSKY/00-00-24.jpg)
![Screenshot at 00:46: The speaker mentions the simple RL recipe used: scaling a 1.5B parameter model with a simple RL recipe.](https://ss.rapidrecap.app/screens/9VFAFrWYSKY/00-00-46.jpg)
![Screenshot at 01:54: Visual confirmation of performance metrics showing the simple method is stable and yields a jump in performance right out of the gate.](https://ss.rapidrecap.app/screens/9VFAFrWYSKY/00-01-54.jpg)
![Screenshot at 07:32: A chart comparison showing the JustRL \(simple\) approach achieving 54.87% accuracy, significantly better than the baseline's 53.01%.](https://ss.rapidrecap.app/screens/9VFAFrWYSKY/00-07-32.jpg)