UI-Venus-1.5 Technical Report

Quick Overview

The UI-Venus-1.5 agent successfully bridges the gap between human intent and computer execution by using a visual-only approach, achieving a 76% success rate on the Android benchmark while avoiding the complexity and potential errors associated with parsing underlying code or XML structures.

Key Points: UI-Venus-1.5 achieved a 76% success rate on the Android benchmark, significantly outperforming previous models like MAI-UI and MAI-UI 3. The agent is designed to operate visually, interpreting screen elements like pixels and button locations just as a human would, rather than parsing underlying code or XML. This visual-only approach bypasses the common issue in older methods where models were heavily rewarded for following procedural steps rather than achieving the actual user goal (e.g., booking a flight). The report highlights that the training heavily focused on real-world optimization, specifically teaching the model to refuse unsafe or unethical instructions and to understand the consequences of actions. The model's architecture is a unified 8B parameter model, merging the strengths of specialized models (like those for vision or code) into a single, more robust agent. A key limitation noted is that the agent must play out the entire task sequence to receive a reward signal, which can introduce latency and computational expense.

Context: This technical report discusses the performance and architecture of the UI-Venus-1.5 agent, developed by the Venus team at Ant Group, which specializes in end-to-end agents capable of interacting with graphical user interfaces. The discussion contrasts this new approach with older models that relied on analyzing underlying code structures (like XML) or explicit step-by-step instructions, emphasizing the importance of visual understanding and real-world context for reliable agent deployment.

Detailed Analysis

The UI-Venus-1.5 agent represents a significant pivot in end-to-end AI agents, focusing on visual interaction rather than parsing underlying code or structured data like XML. Developed by the Venus team at Ant Group, this agent achieved a 76% success rate on the Android benchmark, outperforming previous models like MAI-UI 3 and MAI-UI by a significant margin. The core innovation is its ability to 'see' the screen, interpret pixel coordinates for interactive elements (like buttons or menus), and understand the consequences of actions based on visual feedback, essentially mimicking human visual processing. This procedural knowledge (how to interact) is explicitly trained, allowing the agent to execute complex multi-step tasks like booking flights or shopping online without needing explicit code instructions. The training emphasizes safety by heavily penalizing actions that lead to unsafe, unethical, or impossible outcomes, ensuring real-world viability. A notable trade-off is that the agent must execute the entire task sequence to receive a final reward signal, which can lead to latency and high computational cost. However, the resulting unified 8B parameter model is suggested to be more robust than previous specialized models by merging vision and language skills into a single architecture, offering a simpler path for deploying agents on local hardware.