UI-Venus-1.5 Technical Report

Quick Overview

The UI-Venus-1.5 multimodal large language model represents a fundamental shift from text-based processing to a unified end-to-end system capable of perceiving and interacting with visual information like a human user, achieving a 77.6% success rate on the Android world task, significantly outperforming the previous 72B parameter model, which had a 34% success rate on the same task due to its reliance on a fragmented, non-unified architecture.

Key Points: UI-Venus-1.5 is a new multimodal large language model developed by the Ant Group news team, designed to navigate the digital world by perceiving and interacting with screens visually, much like a human user. It achieved a 77.6% success rate on the Android world task, a massive efficiency jump compared to the previous 72B parameter model's 34% success rate on the same task. The model uses a unified end-to-end architecture, contrasting with older systems that relied on fragmented, modular approaches requiring separate code bridges for different functions. A key feature is its ability to skip cumbersome visual navigation steps (like clicking menus) by using hotkeys or directly acting on visual elements, leading to faster task completion. The framework uses a multi-stage training pipeline, starting with mid-training (Knowledge Injection) and moving to offline reinforcement learning (Offline RL) to ensure high precision across all steps. The explicit design choice to reject the legacy XML-based approach, which often failed when visual elements changed, validates the need for unified, perception-based agents. The success of UI-Venus-1.5 suggests a maturation of AI technology, moving from simple task completion to complex, real-world workflow execution with high reliability.

Context: This technical report analyzes UI-Venus-1.5, a new multimodal large language model developed by the Ant Group news team. The model marks a significant architectural shift in AI, moving away from relying on explicit code parsing (like XML) to directly perceiving and interacting with graphical user interfaces (GUIs) on screens, similar to how a human operates a device.