Dream-VL & Dream-VLA: Open VL and VLA Models with Diffusion Language Model Backbone

Quick Overview

DreamVL and DreamVLA models introduce a fundamental shift in building AI for robotics by replacing the error-prone auto-regressive backbone with a Diffusion Language Model (DLM) backbone, which naturally facilitates long-horizon planning, bidirectional attention, and parallel generation, leading to state-of-the-art performance, including a 27x speedup in generating action chunks for low-level control.

Key Points: The primary bottleneck in complex robotic planning is addressed by replacing the auto-regressive (AR) backbone, which suffers from error accumulation due to its sequential nature, with a diffusion language backbone (DLM). Diffusion models improve planning by starting with noise and iteratively denoising the entire sequence, enabling global coherence and allowing the model to look both backward and forward via bidirectional attention during refinement. DreamVL, the vision-language model baseline, performed competitively against top-tier open data AR models like Mammoth HVL on general benchmarks, proving the DLM architecture is a viable VLM foundation. In high-level planning tasks using the Household benchmark, DreamVL clearly outperformed its AR counterpart, demonstrating that the global reasoning capabilities of diffusion translate to smarter strategic planning. For low-level planning, DreamVL showed extreme robustness; while AR models cratered when forced to predict 12 steps ahead, DreamVL's performance remained stable and actually improved as the action chunk size grew. The most dramatic result was a 27x speed up achieved by generating a chunk of 12 actions using only a single diffusion step, drastically improving latency for real-time control. The fully embodied model, DreamVLA, achieved a new state-of-the-art 97.2% average success rate on the Libero benchmark and a 71.4% success rate on real-world tasks using a Widow X robot.

Context: The video analyzes the paper introducing DreamVL and DreamVLA, which proposes a radical architectural change for embodied AI and robotics planning by moving away from the standard auto-regressive (AR) paradigm. AR models struggle with long horizon planning because they predict actions sequentially, leading to cascading errors, a problem the researchers solve by adopting a diffusion model backbone, similar to those used in image generation, to construct action sequences.