CUA-Skill: Develop Skills for Computer Using Agent

Quick Overview

The CUA-Skill approach, which integrates procedural skills like navigating UI elements with reasoning abilities, successfully allowed an agent to perform complex tasks across 17 Windows applications, achieving a 57.5% success rate on the Atomic Skill benchmark, significantly outperforming the 37.4% success rate of the baseline operator agent on the same tasks.

Key Points: The CUA-Skill agent achieved a 57.5% success rate on the Atomic Skill benchmark, outperforming the baseline operator agent's 37.4% success rate. CUA-Skill involves breaking down tasks into three components: Skill Cell (procedural knowledge), Execution Graph (planning), and Composition Graph (logic). The agent successfully performed actions across 17 common Windows applications, including Excel, Chrome, VS Code, and System Settings. The success of CUA-Skill lies in its ability to perform precise, low-level actions (like pixel-perfect clicks) while maintaining high-level reasoning. The agent was able to perform tasks like changing a font color in PowerPoint using keyboard shortcuts (Alt+H, then F, then C) and was able to execute complex sequences like creating a new folder in File Explorer. The paper suggests that the agent's ability to learn and execute precise UI interactions prevents failures like agents clicking the wrong button or getting stuck in visual feedback loops. The success rate dropped significantly to 11.7% when using a less capable model (Qwen 3 VLL) for the same tasks, highlighting the importance of the CUA-Skill architecture.

Context: This video discusses a new methodology developed by Microsoft Research called CUA-Skill (Cognitive Understanding and Action Skill) aimed at improving the reliability and capability of AI agents when interacting with complex graphical user interfaces (GUIs) in real-world applications. The core concept contrasts the traditional approach, which relies heavily on generalized vision models, with a structured framework that combines procedural knowledge (the Skill Cell) with planning and execution graphs, allowing agents to execute low-level, precise actions reliably.