Video Generation Models Are Good Latent Reward Models

Quick Overview

The paper introduces PRF-L, a novel latent reward model framework that significantly improves video generation quality by leveraging a query-based temporal aggregation mechanism, leading to a 40-50% training speedup and better motion coherence compared to previous models like VLM by distilling knowledge from a pre-trained VGM.

Key Points: PRF-L is a new latent reward model framework that addresses the motion quality and structure issues in existing video generation models. The core innovation involves using a learnable query vector to aggregate temporal information across the entire video timeline, overcoming the noise bottleneck of sampling only early frames. PRF-L achieves a 40-50% training speedup compared to models like VLM by utilizing a frozen, pre-trained VGM backbone for its latent space. The model successfully extracts both structural and motion-related quality signals from the latent space, leading to more physically plausible and coherent video generation. Human evaluation showed PRF-L significantly outperformed older methods in structural and motion quality metrics, achieving a 6.6% gain in quality accuracy over the baseline. The framework successfully solves the issues of poor motion coherence and noise sensitivity present in previous state-of-the-art models.

Context: The discussion centers on the challenges facing video generation models, specifically the difficulty in ensuring high temporal consistency and realistic motion across the entire generated video sequence. Traditional methods often struggle with noise accumulation or focus too heavily on early frames, leading to artifacts later on. The paper introduces PRF-L (Latent Reward Model Framework) as a solution designed to provide better supervision across the entire video structure.

Detailed Analysis

The video discusses the significant challenges in video generation models, particularly generating coherent motion and structure across long sequences. The speaker introduces the PRF-L (Latent Reward Model Framework) paper, which proposes a radical alternative to existing methods by shifting the entire optimization process out of the pixel space and into a latent space. PRF-L leverages a frozen, pre-trained Video Generation Model (VGM) backbone to operate in its latent space, inheriting its structural understanding. The key mechanism is query-based temporal aggregation, where a learnable query vector is used to gather information from the entire video timeline (all time steps) rather than just relying on early or noisy frames. This approach successfully extracts both structural and motion coherence signals. The authors conducted an ablation study demonstrating that PRF-L significantly outperforms older methods, achieving a 40-50% training speedup and a 6.6% gain in quality accuracy over the baseline VLM model in human evaluations. The primary benefit is that PRF-L avoids the performance degradation seen in older models when dealing with high frame count videos or when noise is present in intermediate training steps. The framework solves the problem of models getting 'stuck in bad physics' or noise by providing precise, structure-aware guidance throughout the entire generation process.