# How costly is post-training of LLMs | Lex Fridman Podcast

Source: https://www.youtube.com/watch?v=rg8lMj0Ftzw
Recap page: https://rapidrecap.app/video/rg8lMj0Ftzw
Generated: 2026-02-08T01:33:16.256+00:00

---
## Quick Overview

The cost of post-training LLMs using Reinforcement Learning from Human Feedback (RLHF) is significantly higher than pre-training, with research suggesting that scaling RL requires more than 10 times the training compute compared to the initial pre-training phase, leading to a cost of tens of thousands of dollars per experiment, which creates a barrier for academic researchers.

**Key Points:**
- Scaling Reinforcement Learning (RL) for LLMs requires more than 10 times the training compute compared to the initial pre-training phase (0:48:39).
- The cost of a single RLHF experiment can reach tens of thousands of dollars, making it inaccessible for the average academic researcher (6:55).
- The talk contrasts different reward model types: Reward Models, Outcome Reward Models, Process Reward Models, and Value Functions (3:45).
- Process Reward Models (PRMs) are favored because they assign scores/rewards to intermediate steps, enabling better reasoning supervision compared to just outcome-based methods (3:47).
- The scaling plot shown for AIME (2022/2023) performance indicates a linear increase in evaluation scores when compute is scaled logarithmically for RL-trained models (5:49).
- The speaker suggests that the community will likely follow the scaling paradigm established by the RLHF methods (6:14).
- The cost disparity between pre-training and post-training (RLHF) is a major current hurdle in LLM research (5:00).

![Screenshot at 5:47: A slide titled "Scaling RL: 'more than 10x the training compute than o1'" displays a log-log plot showing the 'Pass @1' performance of OpenAI's models \(o1 to o3\) increasing linearly with exponentially scaled Compute, highlighting the massive compute increase needed for RL scaling.](https://ss.rapidrecap.app/screens/rg8lMj0Ftzw/00-05-47.jpg)

**Context:** The video features a discussion between Lex Fridman and a guest (likely a researcher, given the context of the slides) about the computational costs and scaling laws associated with training Large Language Models (LLMs), specifically focusing on the Reinforcement Learning from Human Feedback (RLHF) post-training phase. The conversation references academic papers and charts detailing how compute scales relative to performance gains in RL for LLMs, contrasting the resource requirements of different reward modeling techniques.

## Detailed Analysis

The discussion centers on the high computational cost of post-training LLMs using Reinforcement Learning from Human Feedback (RLHF). The guest notes that scaling RL requires more than ten times the compute used for initial pre-training, citing a seminal Meta paper on scaling RL. This expense, estimated at tens of thousands of dollars per experiment, creates a significant barrier for academic researchers. The conversation clarifies the differences between four reward model classes: Reward Models (predicting text quality), Outcome Reward Models (predicting final success/failure), Process Reward Models (rewarding intermediate steps), and Value Functions (expected return). Process Reward Models are highlighted as superior for complex reasoning tasks because they supervise intermediate steps, unlike outcome-based methods. A chart illustrates that for the AIME benchmark, performance gains scale linearly with compute when plotted logarithmically, but the compute required for the RL phase is vastly greater than the initial phase. The speaker concludes that the current scaling paradigm for RLHF will likely continue to be followed, despite the high cost, suggesting that finding ways to reduce this cost is crucial for broader research.

### Computational Cost of RLHF

- Scaling RL requires >10x the training compute of pre-training
- Costs reach tens of thousands of dollars per experiment, posing accessibility issues for academics
- The high cost forces researchers to be more incremental and cautious with experiments (0:04-7:05).

### Types of Reward Models

- Reward Models score text quality at EOS token
- Outcome Reward Models score final success/failure
- Process Reward Models score intermediate steps, better for reasoning
- Value Functions predict expected return (3:45).

### Scaling Laws and Performance

- A graph shows AIME performance scaling linearly with compute (log scale) for RL-trained models, with OpenAI o3 significantly surpassing o1 (5:47).

### Future Outlook

- The current scaling paradigm (heavy compute for RL) is likely to persist
- The community needs to find ways to reduce the cost of RLHF/process supervision to make it more accessible (6:11-7:11).

![Screenshot at 0:03: Lex Fridman sitting at his desk, dressed in a suit, speaking into a microphone.](https://ss.rapidrecap.app/screens/rg8lMj0Ftzw/00-00-03.jpg)
![Screenshot at 0:19: The guest speaking animatedly, gesturing with his hands while explaining computational differences.](https://ss.rapidrecap.app/screens/rg8lMj0Ftzw/00-00-19.jpg)
![Screenshot at 3:40: A slide titled "The Art of Scaling Reinforcement Learning Compute for LLMs" appears, detailing methods like Process Reward Models \(PRMs\).](https://ss.rapidrecap.app/screens/rg8lMj0Ftzw/00-03-40.jpg)
![Screenshot at 3:45: A comparison table detailing four Model Classes \(Reward, Outcome, Process, Value Functions\) based on what they predict, how they are trained, and their LM structure.](https://ss.rapidrecap.app/screens/rg8lMj0Ftzw/00-03-45.jpg)
![Screenshot at 5:47: A chart titled "Validation perf. scaling RL" showing Pass Rate \(log scale\) vs. GPU Hours \(log scale\), illustrating the significant performance gains achieved by scaling compute with RL.](https://ss.rapidrecap.app/screens/rg8lMj0Ftzw/00-05-47.jpg)