On GRPO Collapse in Search-R1: The Lazy Likelihood-Displacement Death Spiral

Quick Overview

The collapse of the GRPO (Group Relative Policy Optimization) reward curve is caused by a self-reinforcing death spiral where the model's initial low confidence in correct actions amplifies errors, leading to instability and eventual failure, despite external reward signals appearing positive.

Key Points: The problem discussed is the 'Lazy Likelihood-Displacement Death Spiral' occurring in GRPO training. The core issue stems from the model's initial low confidence in correct responses, especially when using search tools. GRPO's reward mechanism penalizes the model for low confidence in correct answers, causing the model to learn to avoid correct actions to maintain a non-negative reward signal. This leads to a self-reinforcing progression where the model's internal confidence erodes, causing the reward curve to decay rapidly (a 'death spiral'). The paper suggests a solution involving masking out the final answer token's uncertainty contribution to the reward calculation, specifically for the 'Lazy Likelihood-Displacement' mechanism. The 7B model using the modified reward saw a 37% performance gain over the vanilla model on specific benchmarks, demonstrating the fix's effectiveness. The problem is inherent to the policy; it is not a penalty for bad actions but a systemic flaw in how confidence/uncertainty is incorporated into the reward signal.

Context: This video analyzes a critical instability issue observed during the training of Large Language Models (LLMs) using the Group Relative Policy Optimization (GRPO) algorithm, specifically focusing on a phenomenon termed the 'Lazy Likelihood-Displacement Death Spiral.' This instability arises when the model's internal confidence in its own actions falls too low, causing the reward signal to degrade even when the model is performing correctly, creating a negative feedback loop.

Detailed Analysis

The video explains the 'Lazy Likelihood-Displacement Death Spiral' (LLD) in GRPO training, which is a significant instability issue. The problem originates in Phase 1 when the model generates answers using external tools (like search) but has low internal confidence in its steps. The LLD mechanism, which is part of the reward calculation, penalizes the model for low confidence even in correct intermediate steps. This causes the model to learn to avoid correct, high-confidence actions to maintain a non-negative reward, leading to a self-reinforcing progression where confidence plummets, and the reward curve decays rapidly, ultimately causing the model to fail catastrophically. The solution proposed involves masking out the contribution of the final answer token's uncertainty during reward calculation in the LLD mechanism. This targeted intervention prevents the policy from being overly penalized for uncertainty in the final step, thereby stabilizing the training. Experiments showed that the 7B model, when using this modified reward signal, achieved significant gains (around 37%) on multi-hop benchmarks compared to the vanilla GRPO model, proving that the fix is crucial for achieving stable, high-quality performance.