Dover. Intervention-Driven Auto Debugging For LLM Multi-Agent Systems

Quick Overview

The Dover multi-agent debugging framework successfully moves debugging from a purely analytical, log-based approach to an intervention-driven process, achieving a significant reduction in failure attribution, with successful intervention and verification leading to a 49% to 67% success rate on previously failed multi-step tasks.

Key Points: Dover is an intervention-driven auto-debugging framework for LLM multi-agent systems, proposed in a paper from the Chinese Academy of Sciences and Microsoft. The framework addresses the difficulty of debugging complex multi-step systems by treating the log history as a testable hypothesis rather than just an analysis artifact. Dover flips the script: instead of just analyzing logs, it proposes explicit, targeted interventions (fixes) at specific steps (e.g., step 42) and then verifies if the subsequent execution succeeds. In testing (Case 46), the original system failed to find a file; when Dover intervened at that step, the task succeeded, proving the hypothesis was flawed. The method achieved a high success rate, recovering from failure between 49% and 67% of the time across complex multi-step tasks, significantly outperforming previous methods. A key engineering hurdle overcome was the lack of an intrinsic tool to execute scroll actions, which Dover addressed by implementing a PDF processor and a fully functional checkpoint and replay interface. The success of this method means that future LLM agents will have built-in verification capabilities, preventing the tool from being missing when needed for debugging.

Context: The video discusses a research paper titled "Dover: Intervention-Driven Auto Debugging For LLM Multi-Agent Systems" from the Chinese Academy of Sciences and Microsoft. This work tackles the significant challenge of debugging complex, multi-step LLM agent systems where errors often lead to opaque failure logs. The core concept introduced is shifting debugging from a retrospective analysis of logs to an active, intervention-based verification loop, aiming to build more reliable autonomous systems.