# Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresight

Source: https://www.youtube.com/watch?v=TXmg_TEogC0
Recap page: https://rapidrecap.app/video/TXmg_TEogC0
Generated: 2025-11-25T20:04:39.973+00:00

---
## Quick Overview

The Mantis vision-language-action (VLA) model achieves superior performance in robotic control tasks by decoupling visual foresight (predicting future visual states) from action prediction, utilizing a three-phase training recipe that involves multiple gap vision training, supervised mixed training on video datasets like SSV2, and finally, leveraging the large language model (LLM) backbone for abstract reasoning, which results in significantly better generalization, especially for out-of-domain instructions, compared to other large models like LLaVA.

**Key Points:**
- Mantis is a novel vision-language-action (VLA) model designed to improve robotic control by disentangling visual foresight from action prediction.
- The model uses a three-phase training recipe: multiple gap vision training, supervised mixed training on datasets like SSV2 (human manipulation videos), and leveraging the LLM backbone for abstract reasoning.
- Mantis achieved a 96.7% average success rate on the difficult Libero benchmark, significantly outperforming other large models like LLaVA (which scored 0.5% success on abstract reasoning tasks).
- The disentangled visual foresight mechanism frees up the core model's capacity to focus on reasoning about the physical world rather than just predicting pixels or static background elements.
- The model successfully handles abstract instructions, such as translating 'put the apple in the bowl' into a precise physical action, demonstrating strong generalization.
- The training approach ensures the model learns the physics of how things move in the world, rather than just pattern-matching language to visual states.

![Screenshot at 04:23: The speaker points out that the key benefit of Mantis is that decoupling the visual foresight \(predicting the future state\) from the action prediction allows the model to focus on abstract reasoning, confirmed by the visual representation of the three-stage training process.](https://ss.rapidrecap.app/screens/TXmg_TEogC0/00-04-23.png)

**Context:** This video discusses the Mantis model, a new architecture in the field of vision-language-action (VLA) models aimed at enhancing robotic control capabilities. The core innovation of Mantis is its approach to breaking down complex decision-making into separate components: predicting what the future will look like (visual foresight) and deciding what action to take next (action prediction). This architecture contrasts with traditional models that often conflate these two processes, leading to limitations when faced with novel or abstract commands.

## Detailed Analysis

The discussion focuses on the Mantis model, a Versatile Vision-Language-Action (VLA) model, highlighting its novel architecture that separates visual foresight from action prediction. The main takeaway is that this disentanglement leads to superior performance, especially in generalization. The model employs a three-phase training recipe. Stage one involves multiple gap vision training. Stage two uses supervised mixed training on human manipulation video datasets like SSV2. Stage three leverages the large language model (LLM) backbone for abstract reasoning. This process results in the model learning the physics of movement rather than just matching language to visual states. The authors note that this approach allows the core model to focus on reasoning rather than getting bogged down in predicting every pixel change. When tested against the difficult Libero benchmark, Mantis achieved a 96.7% average success rate, significantly outperforming models like LLaVA, which struggled with abstract reasoning tasks. The success rate is attributed to the model's ability to handle abstract instructions (like 'put the apple in the bowl') and translate them into precise physical actions, confirming that the decoupled foresight was the correct architectural move.

### Mantis Model Overview

- Novel architecture separates visual foresight from action prediction
- Aims to solve fundamental optimization problems in robotic control
- Achieved 96.7% success on the Libero benchmark

### Three-Phase Training Recipe

- Stage 1: Multiple gap vision training
- Stage 2: Supervised mixed training using SSV2 human manipulation videos
- Stage 3: Leveraging LLM backbone for abstract reasoning

### Key Advantages

- Decoupling frees up core model capacity for reasoning over pure prediction
- Superior generalization to out-of-domain instructions compared to LLaVA
- Model learns physical world dynamics rather than just visual pattern matching

### Testing and Results

- Tested against Libero benchmark where LLaVA scored 0.5% on abstract reasoning
- Mantis successfully handled abstract commands like 'put the bear on number 8'
- Demonstrated ability to perform nuanced tasks requiring real-world understanding

![Screenshot at 00:00: Initial splash screen for the podcast/video featuring two people at microphones with the text 'BECOME A MEMBER TODAY!'](https://ss.rapidrecap.app/screens/TXmg_TEogC0/00-00-00.png)
![Screenshot at 04:08: Visual slide illustrating the three stages of the Mantis training process: Stage 1 \(Multiple Gap Vision Training\), Stage 2 \(Vision-Action Mixed Training\), and Stage 3 \(LLM-based Reasoning\).](https://ss.rapidrecap.app/screens/TXmg_TEogC0/00-04-08.png)
![Screenshot at 04:43: Visual showing the concept of 'target patches' being used to focus the model's attention on the most relevant parts of the image for reasoning.](https://ss.rapidrecap.app/screens/TXmg_TEogC0/00-04-43.png)
![Screenshot at 07:34: Graphic representation of the Libero benchmark testing where Mantis outperformed other models in abstract reasoning tasks.](https://ss.rapidrecap.app/screens/TXmg_TEogC0/00-07-34.png)
![Screenshot at 09:10: A dynamic graph/waveform illustrating the difference in performance metrics, showing Mantis's improved success rate compared to baseline models.](https://ss.rapidrecap.app/screens/TXmg_TEogC0/00-09-10.png)