Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresight

Quick Overview

The Mantis vision-language-action (VLA) model achieves superior performance in robotic control tasks by decoupling visual foresight (predicting future visual states) from action prediction, utilizing a three-phase training recipe that involves multiple gap vision training, supervised mixed training on video datasets like SSV2, and finally, leveraging the large language model (LLM) backbone for abstract reasoning, which results in significantly better generalization, especially for out-of-domain instructions, compared to other large models like LLaVA.

Key Points: Mantis is a novel vision-language-action (VLA) model designed to improve robotic control by disentangling visual foresight from action prediction. The model uses a three-phase training recipe: multiple gap vision training, supervised mixed training on datasets like SSV2 (human manipulation videos), and leveraging the LLM backbone for abstract reasoning. Mantis achieved a 96.7% average success rate on the difficult Libero benchmark, significantly outperforming other large models like LLaVA (which scored 0.5% success on abstract reasoning tasks). The disentangled visual foresight mechanism frees up the core model's capacity to focus on reasoning about the physical world rather than just predicting pixels or static background elements. The model successfully handles abstract instructions, such as translating 'put the apple in the bowl' into a precise physical action, demonstrating strong generalization. The training approach ensures the model learns the physics of how things move in the world, rather than just pattern-matching language to visual states.

Context: This video discusses the Mantis model, a new architecture in the field of vision-language-action (VLA) models aimed at enhancing robotic control capabilities. The core innovation of Mantis is its approach to breaking down complex decision-making into separate components: predicting what the future will look like (visual foresight) and deciding what action to take next (action prediction). This architecture contrasts with traditional models that often conflate these two processes, leading to limitations when faced with novel or abstract commands.