Planning with Sketch-Guided Verification for Physics-Aware Video Generation

Quick Overview

The Sketch-Guided Verification (SketchVerify) framework significantly improves the efficiency and physical realism of physics-aware video generation by using a lightweight proxy to check motion trajectories against physical constraints, achieving a 10x speedup over the costly diffusion-based approach while maintaining high accuracy.

Key Points: SketchVerify reduces generation time for physics-aware video from potentially an hour to about 5 minutes by decoupling the planning/verification step from the resource-intensive rendering step. The method uses a lightweight, low-fidelity video proxy to check if the generated motion (e.g., a ball falling) adheres to basic physics like gravity and momentum. The core improvement is decoupling the planning/verification from the high-fidelity diffusion backbone, allowing the generator to focus only on photorealism once the motion is verified. In ablation studies, SketchVerify improved physics coherence accuracy by 6% compared to methods relying solely on language verification against the underlying diffusion model. The verification process checks for violations of physical laws (like momentum or gravity) and structural integrity in the motion trajectory. The framework successfully ensures that generated actions, such as a basketball shot, follow a plausible parabolic arc rather than impossible movements like hovering or instantly changing direction.

Context: This video discusses a novel method called Sketch-Guided Verification (SketchVerify) designed to address the challenges in AI video generation, specifically where large generative models often fail to adhere to fundamental physical laws, resulting in unrealistic or impossible motions. The research proposes a three-step pipeline—planning, verification, and generation—to make the process more efficient and physically consistent, especially for complex interactions.

Detailed Analysis

The video details the SketchVerify framework, which tackles the problem of physics incoherence in AI video generation. The framework operates in three steps: planning, verification, and generation. Step one involves planning and scene parsing, where the system uses a high-level planning model (like Gemini 2.5) to decompose a complex instruction into smaller, actionable sub-instructions, such as 'dribble forward, then stop, then jump and shoot the ball towards the basket.' The second step, verification, is where SketchVerify excels. Instead of immediately rendering a high-fidelity, expensive video for every step, the system generates a cheap, low-fidelity proxy video that accurately depicts the object's movement over time. A verifier then checks this proxy against physical laws (like gravity and inertia) using structural prompts. If the motion violates physics (e.g., an object floats unnaturally or violates momentum conservation), the trajectory is rejected. This verification significantly reduces the computational load. The third step is generation, where the diffusion model only renders the high-fidelity video once the motion plan has been verified as physically plausible. The researchers found that using SketchVerify improved physics coherence accuracy by 6% over language-only verification methods when tested against benchmarks that include complex physical scenarios like car crashes or water splashes. This iterative refinement process—planning, verifying the proxy, and then rendering—is far more efficient than brute-force attempts, saving significant GPU time.