Graph-Based Exploration for ARC-AGI-3 Interactive Reasoning Tasks

Quick Overview

The ARC-AGI-3 benchmark demonstrates that LLMs, even large ones, struggle with interactive reasoning tasks that require systematic exploration and state tracking, as evidenced by the failure of the graph-based agent to solve tasks that required tracking both explicit graph structure and the agent's own history, leading to significant performance gaps compared to human performance.

Key Points: The ARC-AGI-3 benchmark tests interactive reasoning, revealing that current LLMs struggle with tasks requiring systematic exploration and state tracking. The graph-based agent failed to solve tasks requiring both explicit graph structure knowledge and tracking of its own history, leading to an infinite failure loop. The baseline LLM+DSL approach scored 0% on initial preview tasks, while the graph-based exploration agent scored 0% on the initial 100% of the preview tasks, highlighting the difficulty. The successful approach involved combining the LLM's knowledge with a structured graph representation, allowing for systematic exploration and state management. The structure ensures that the agent's actions are grounded in an explicit, trackable state, preventing looping caused by poor memory or exploration. The performance jump was substantial: the structured approach solved 19 levels in 8 hours, whereas the baseline LLM only solved 5 private levels, demonstrating the necessity of structure. The core challenge is not just coding ability but maintaining a reliable external mechanism for state tracking across complex, multi-step interactions.

Context: This video discusses a new research paper introducing the ARC-AGI-3 benchmark, which focuses on interactive reasoning tasks that challenge AI agents to navigate complex, dynamic environments. The core concept explored is the difference between models trained only on vast datasets (like large language models) and those that incorporate explicit structural guidance, specifically graph representations, to manage state and exploration during problem-solving.