How AI Coding Agents Communicate: A Study of PR Descriptions and Human Review Responses

Quick Overview

The study reveals that AI coding agents like GitHub Copilot and Claude exhibit significantly different approaches to generating code descriptions and responding to human reviews, with Copilot producing highly structured, concise summaries and high sentiment, while Claude produces verbose, less structured text that occasionally triggers negative sentiment, despite both agents demonstrating high overall code quality.

Key Points: GitHub Copilot generated descriptions that were highly structured, concise, and achieved the highest positive sentiment score (82.6% merge rate). Claude generated verbose, less structured text, often including fluff words, leading to a lower sentiment score and higher human review time compared to Copilot. The study compared AI agents' performance in generating PR descriptions and handling human review responses based on data from 33,600 pull requests. The most provocative data point showed that while both agents generated correct code, Copilot's descriptions were perceived as more useful, while Claude's were perceived as overly polite and less informative, leading to more human follow-up questions. The speed difference was significant: Copilot completed the entire process in about 1 hour 12 seconds on average, whereas Claude took about 1 hour 13 minutes. The researchers suggest that future AI agents must learn to squash small commits and avoid dumping raw code blocks in descriptions, prioritizing structure over verbosity.

Context: This video analyzes and compares the communication styles of two leading AI coding agents, GitHub Copilot and Claude, specifically focusing on how they generate Pull Request (PR) descriptions and how human reviewers respond to them. The analysis is based on a large dataset of 33,600 pull requests, aiming to determine which agent's output leads to more efficient and positive human-AI collaboration in software engineering.

Detailed Analysis

The analysis compares GitHub Copilot and Claude in generating PR descriptions and handling human reviews across 33,600 pull requests. GitHub Copilot consistently outperformed Claude in generating concise, well-structured descriptions that resulted in fast human approvals (1 hour 1 minute completion time). Copilot's PR descriptions, while concise, generated the highest positive sentiment score (82.6% merge rate) because they focused on specific, relevant information and avoided unnecessary detail. Conversely, Claude generated verbose, often fluffy text, which led to longer review times (1 hour 13 minutes average completion time) and lower sentiment. The researchers observed that Claude often included polite filler words like "please," "thank you," and "kindly," which masked the actual changes, leading to reviewer frustration. For example, when submitting a fix, Claude provided a lengthy narrative, whereas Copilot provided a direct summary. Furthermore, Claude's output generated the highest volume of negative sentiment comments, suggesting human reviewers perceived its verbosity as noise or even hostility, despite the code being correct. The study concludes that structure and conciseness, as demonstrated by Copilot, are superior to excessive politeness or verbosity when reviewing technical code changes. The core takeaway is that in a highly technical environment, clarity and efficiency outweigh unnecessary politeness, as evidenced by Copilot's superior merge rate and speed.