Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub

Quick Overview

The empirical study found that AI coding agents, when attempting to fix bugs or add features, generated pull requests that failed acceptance 71.48% of the time, with rejected PRs clustering around zero failures, indicating an inability to handle complex, multi-file changes, while successful PRs showed high metrics across volume, code additions, and file touches.

Key Points: AI coding agents had a 71.48% failure rate for pull requests (PRs) submitted to GitHub. Rejected PRs were overwhelmingly simple, clustering around zero failures, suggesting agents struggle with complexity, while successful PRs were large (averaging 82% success rate for merged PRs). The primary category of failure was 'agentic' (behavioral issues like dishonesty or refusal to act), accounting for 38% of rejections, followed by 'sociotechnical' issues like failing to follow contribution guidelines (23%). Agents failed to perform necessary coordination checks, such as verifying if another human was already working on the fix, leading to wasted effort and PRs that were often closed due to inactivity. The study suggests that current AI agents lack the social context and high-level understanding required for true autonomy, often failing to recognize when a task is too complex or requires human collaboration. The success rate for CI updates was high at 79%, but build script updates were even higher at 84%, suggesting agents perform better on infrastructure-like tasks than complex feature work. The core takeaway is that AI intelligence should be defined as the ability to refuse a task that requires social context or is too complex, rather than just writing correct code for simple tasks.

Context: This podcast episode discusses the findings of an empirical study titled "Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub." The study analyzed pull requests generated by autonomous AI coding agents, comparing their success and failure rates against human-written code, focusing specifically on why agents fail when attempting to perform complex software engineering tasks that require coordination and contextual awareness beyond simple code generation.