How costly is post-training of LLMs | Lex Fridman Podcast

Quick Overview

The cost of post-training LLMs using Reinforcement Learning from Human Feedback (RLHF) is significantly higher than pre-training, with research suggesting that scaling RL requires more than 10 times the training compute compared to the initial pre-training phase, leading to a cost of tens of thousands of dollars per experiment, which creates a barrier for academic researchers.

Key Points: Scaling Reinforcement Learning (RL) for LLMs requires more than 10 times the training compute compared to the initial pre-training phase (0:48:39). The cost of a single RLHF experiment can reach tens of thousands of dollars, making it inaccessible for the average academic researcher (6:55). The talk contrasts different reward model types: Reward Models, Outcome Reward Models, Process Reward Models, and Value Functions (3:45). Process Reward Models (PRMs) are favored because they assign scores/rewards to intermediate steps, enabling better reasoning supervision compared to just outcome-based methods (3:47). The scaling plot shown for AIME (2022/2023) performance indicates a linear increase in evaluation scores when compute is scaled logarithmically for RL-trained models (5:49). The speaker suggests that the community will likely follow the scaling paradigm established by the RLHF methods (6:14). The cost disparity between pre-training and post-training (RLHF) is a major current hurdle in LLM research (5:00).

Context: The video features a discussion between Lex Fridman and a guest (likely a researcher, given the context of the slides) about the computational costs and scaling laws associated with training Large Language Models (LLMs), specifically focusing on the Reinforcement Learning from Human Feedback (RLHF) post-training phase. The conversation references academic papers and charts detailing how compute scales relative to performance gains in RL for LLMs, contrasting the resource requirements of different reward modeling techniques.

Detailed Analysis

The discussion centers on the high computational cost of post-training LLMs using Reinforcement Learning from Human Feedback (RLHF). The guest notes that scaling RL requires more than ten times the compute used for initial pre-training, citing a seminal Meta paper on scaling RL. This expense, estimated at tens of thousands of dollars per experiment, creates a significant barrier for academic researchers. The conversation clarifies the differences between four reward model classes: Reward Models (predicting text quality), Outcome Reward Models (predicting final success/failure), Process Reward Models (rewarding intermediate steps), and Value Functions (expected return). Process Reward Models are highlighted as superior for complex reasoning tasks because they supervise intermediate steps, unlike outcome-based methods. A chart illustrates that for the AIME benchmark, performance gains scale linearly with compute when plotted logarithmically, but the compute required for the RL phase is vastly greater than the initial phase. The speaker concludes that the current scaling paradigm for RLHF will likely continue to be followed, despite the high cost, suggesting that finding ways to reduce this cost is crucial for broader research.