# AIDev: Studying AI Coding Agents on GitHub Source: https://www.youtube.com/watch?v=qH40bRVqMS0 Recap page: https://rapidrecap.app/video/qH40bRVqMS0 Generated: 2026-02-20T12:03:14.164+00:00 --- ## Quick Overview The research paper "AIDev: Studying AI Coding Agents on GitHub" details the creation and evaluation of the SE 3.0 framework, which shifts the view of AI coding agents from simple tools to collaborative teammates by analyzing nearly a million GitHub pull requests, finding that agents are currently more effective at writing new code than fixing existing code, which creates a significant liability and forces humans to focus on reviewing the AI's complex, but often flawed, output. **Key Points:** - The research introduces the SE 3.0 framework to study AI coding agents acting as teammates rather than just tools. - The dataset used comprises nearly one million pull requests from GitHub, including metadata like timestamps and merge/rejection status. - The study found that AI agents are currently better at generating new code than fixing bugs or refactoring existing code, posing a security risk. - The quality metric identified as most important for the long-term health of the industry is ecological validity—how the AI performs in real-world scenarios. - The paper analyzed 29,000 human review verdicts, noting that poor code submissions from AI often lead to human reviewers spending significant time fixing them. - The trend suggests developers are shifting from being creators to maintainers of AI-generated spaghetti code, which increases technical debt. - The ultimate goal of the research is to shift the scientific conversation from 'Can the AI solve this puzzle?' to 'How do humans and AI effectively collaborate?' ![Screenshot at 00:00: The opening screen displays the title 'Become A Member Today!' over an image of two podcasters, indicating this is content from the 'AI Papers Podcast Daily' channel, which sets the stage for discussing a new research paper.](https://ss.rapidrecap.app/screens/qH40bRVqMS0/00-00-00.jpg) **Context:** The video discusses findings from a research paper titled "AIDev: Studying AI Coding Agents on GitHub," which examines the evolving role of AI coding assistants like GitHub Copilot and Cursor in software development workflows. The researchers argue that the industry is moving past treating AI as a simple spell-checker or autocomplete tool and toward viewing these agents as genuine collaborators, necessitating a shift in how developers interact with and validate AI-generated code. ## Detailed Analysis The research discussed in the video, titled "AIDev: Studying AI Coding Agents on GitHub," introduces the SE 3.0 framework, which reframes the role of AI coding agents from mere tools to actual teammates. This framework is based on analyzing a massive dataset of nearly one million GitHub pull requests, including metadata on human reviews, such as whether a pull request was merged or rejected. The key finding is that AI agents excel at generating new code but struggle with fixing existing, complex codebases, leading to what researchers call 'automation bias' where humans implicitly trust the AI's output without adequate scrutiny. This reliance on AI for generating code, even when flawed, increases technical debt and forces human developers (who should be architects and security analysts) into the role of cleaning up AI-generated messes. The paper explicitly tests the agents against human reviewers on bug fixes and finds that AI-generated fixes often introduce new vulnerabilities or are simply incorrect, forcing the human reviewer to spend time correcting them instead of focusing on high-level architecture. The researchers stress that the most critical metric for the long-term health of the industry is 'ecological validity'—how the AI performs in real-world, messy environments, rather than just in sterile lab tests. The overall conclusion is that the workflow is changing; developers risk becoming mere maintainers of complex, low-quality AI code if they do not maintain strong foundational knowledge and rigorous testing practices. ### Key Research Focus - Shift from AI as a tool to AI as a teammate - Framework introduced is SE 3.0 - Goal is to study collaboration, not just generation ### Dataset and Scale - Dataset contains nearly one million GitHub pull requests - Includes metadata like timestamps, merge/rejection status, and human review comments ### AI Performance Analysis - AI agents are better at writing new code than fixing bugs or refactoring old code - Code generated can look perfect superficially but be logically wrong underneath ### Reviewer Behavior - Human reviewers spend significant time fixing poor AI submissions, leading to an increased workload and security vulnerabilities (e.g., exposing keys) ### Industry Concerns - Risk of developers becoming maintainers of messy AI code (spaghetti code) rather than creators - Technical debt increases when AI generates code without proper inspection ### Critical Metric - Ecological validity—how AI performs in real-life messy situations—is the most important long-term health metric, not benchmarks. ### Conclusion - The conversation must move from 'Can AI solve this?' to 'How do humans and AI effectively collaborate?' ![Screenshot at 00:00: The initial visual displays the podcast branding and an invitation to 'Become A Member Today!' over an image of two podcasters, introducing the content source.](https://ss.rapidrecap.app/screens/qH40bRVqMS0/00-00-00.jpg) ![Screenshot at 00:14: A speaker comments that version numbers like '3.0' are often just marketing, setting a critical tone for evaluating the new research.](https://ss.rapidrecap.app/screens/qH40bRVqMS0/00-00-14.jpg) ![Screenshot at 00:39: The speaker emphasizes the need to understand the shift from AI as a tool to AI as an agent, referencing the significant new research paper.](https://ss.rapidrecap.app/screens/qH40bRVqMS0/00-00-39.jpg) ![Screenshot at 01:57: A specific data point is highlighted: the dataset contains nearly one million pull requests, underscoring the massive scale of the study.](https://ss.rapidrecap.app/screens/qH40bRVqMS0/00-01-57.jpg) ![Screenshot at 05:50: The visual shows the two podcasters, illustrating the discussion about the human reviewer's role in checking AI-generated code.](https://ss.rapidrecap.app/screens/qH40bRVqMS0/00-05-50.jpg)