# On GRPO Collapse in Search-R1: The Lazy Likelihood-Displacement Death Spiral

Source: https://www.youtube.com/watch?v=k1St79ykkg8
Recap page: https://rapidrecap.app/video/k1St79ykkg8
Generated: 2025-12-26T20:03:35.978+00:00

---
## Quick Overview

The collapse of the GRPO (Group Relative Policy Optimization) reward curve is caused by a self-reinforcing death spiral where the model's initial low confidence in correct actions amplifies errors, leading to instability and eventual failure, despite external reward signals appearing positive.

**Key Points:**
- The problem discussed is the 'Lazy Likelihood-Displacement Death Spiral' occurring in GRPO training.
- The core issue stems from the model's initial low confidence in correct responses, especially when using search tools.
- GRPO's reward mechanism penalizes the model for low confidence in correct answers, causing the model to learn to avoid correct actions to maintain a non-negative reward signal.
- This leads to a self-reinforcing progression where the model's internal confidence erodes, causing the reward curve to decay rapidly (a 'death spiral').
- The paper suggests a solution involving masking out the final answer token's uncertainty contribution to the reward calculation, specifically for the 'Lazy Likelihood-Displacement' mechanism.
- The 7B model using the modified reward saw a 37% performance gain over the vanilla model on specific benchmarks, demonstrating the fix's effectiveness.
- The problem is inherent to the policy; it is not a penalty for bad actions but a systemic flaw in how confidence/uncertainty is incorporated into the reward signal.

![Screenshot at 13:13: The speaker identifies the core problem as the 'Lazy Likelihood-Displacement' mechanism creating a fundamental bottleneck in GRPO training, leading to instability.](https://ss.rapidrecap.app/screens/k1St79ykkg8/00-13-13.jpg)

**Context:** This video analyzes a critical instability issue observed during the training of Large Language Models (LLMs) using the Group Relative Policy Optimization (GRPO) algorithm, specifically focusing on a phenomenon termed the 'Lazy Likelihood-Displacement Death Spiral.' This instability arises when the model's internal confidence in its own actions falls too low, causing the reward signal to degrade even when the model is performing correctly, creating a negative feedback loop.

## Detailed Analysis

The video explains the 'Lazy Likelihood-Displacement Death Spiral' (LLD) in GRPO training, which is a significant instability issue. The problem originates in Phase 1 when the model generates answers using external tools (like search) but has low internal confidence in its steps. The LLD mechanism, which is part of the reward calculation, penalizes the model for low confidence even in correct intermediate steps. This causes the model to learn to avoid correct, high-confidence actions to maintain a non-negative reward, leading to a self-reinforcing progression where confidence plummets, and the reward curve decays rapidly, ultimately causing the model to fail catastrophically. The solution proposed involves masking out the contribution of the final answer token's uncertainty during reward calculation in the LLD mechanism. This targeted intervention prevents the policy from being overly penalized for uncertainty in the final step, thereby stabilizing the training. Experiments showed that the 7B model, when using this modified reward signal, achieved significant gains (around 37%) on multi-hop benchmarks compared to the vanilla GRPO model, proving that the fix is crucial for achieving stable, high-quality performance.

### Problem Identification

- The core issue is the 'Lazy Likelihood-Displacement Death Spiral' causing instability in GRPO training
- This stems from low confidence in intermediate steps when using tools like search
- The LLD mechanism penalizes low confidence, creating a negative feedback loop.

### The Solution

- Masking out the uncertainty contribution from the final answer token in the LLD reward calculation
- This prevents the model from being penalized for accurate but uncertain steps.

### Results and Validation

- The modified reward mechanism stabilized training, preventing the catastrophic collapse seen previously
- The 7B model saw a 37% relative improvement in performance on multi-hop benchmarks compared to vanilla GRPO
- The fix is essential for reliable tool use and complex reasoning.

![Screenshot at 00:00: The initial call to action screen for membership.](https://ss.rapidrecap.app/screens/k1St79ykkg8/00-00-00.jpg)
![Screenshot at 01:00: Visual representation of the iterative loop where tool use leads to feedback and self-correction.](https://ss.rapidrecap.app/screens/k1St79ykkg8/00-01-00.jpg)
![Screenshot at 04:45: The speaker outlines the three main phases of the failure mechanism.](https://ss.rapidrecap.app/screens/k1St79ykkg8/00-04-45.jpg)
![Screenshot at 08:38: The speaker discusses the inherent flaw rooted in low confidence trajectories dominating the learning process.](https://ss.rapidrecap.app/screens/k1St79ykkg8/00-08-38.jpg)
![Screenshot at 12:55: The speaker introduces the fix: masking the uncertainty penalty applied to the final answer token.](https://ss.rapidrecap.app/screens/k1St79ykkg8/00-12-55.jpg)