Will It Survive? Deciphering the Fate of AI-Generated Code in Open Source

Quick Overview

AI-generated code, specifically from agents like Copilot, Cursur, and Devin, exhibits superior durability and lower modification rates compared to human-written code in open-source projects, suggesting AI code is statistically less likely to be modified or break than human code, although it may exhibit different types of errors like logical flaws or dependency issues.

Key Points: AI-generated code (from agents like Copilot, Cursur, and Devin) shows a 15% lower modification rate than human-written code over the study period. The study tracked 201 open-source projects, comparing AI-generated code to human-written code, using survival analysis. The AI agent Devin showed a death rate of 71.7% compared to 79.3% for humans, indicating higher long-term stability for Devin's code. The primary reason AI code survives longer appears to be risk aversion (avoiding making changes) rather than superior quality, as demonstrated by a lower rate of corrective modifications (26.3% vs 23% for humans). AI code, while syntactically correct, often exhibits logical flaws or misses necessary steps like semicolon calls or handling edge cases, which humans typically catch. The temporal prediction model for when code breaks performed no better than random guessing, suggesting timing is hard to predict. The inherent difference in tool selection—AI agents operating closer to a Level 5 autonomous system versus human developers—contributes to the difference in code survival.

Context: This video analyzes a research paper titled "Will It Survive? Deciphering the Fate of AI-Generated Code in Open Source," conducted by researchers from Concordia University, including Musfika Rahman and Shihab. The study challenges the common narrative that AI-generated code is inherently disposable by quantitatively measuring the survival rate of code written by various AI agents (like Copilot, Cursur, and Devin) against human-written code in real-world open-source projects.

Detailed Analysis

The research examined 201 open-source projects to determine the survival rate of AI-generated code versus human-written code, employing survival analysis methods borrowed from medical research. The key finding is that AI-generated code is statistically more durable, showing a 15 percentage point lower rate of modification than human code. Specifically, Devin's code had a lower death rate (71.7%) compared to human code (79.3%) by the end of the study period (December 31, 2025). The researchers hypothesize this durability stems from risk aversion: AI agents, operating more autonomously (like a Level 5 self-driving car), generate code that is often left untouched because developers fear breaking something or lack the deep understanding to confidently refactor it, leading to a 'don't touch it' mentality. While AI code is strong in syntax and rarely produces typos, it frequently contains logical flaws, misses necessary elements like semicolon calls, or fails to account for edge cases—issues humans typically correct. Furthermore, predicting the exact timing of when AI code breaks proved impossible, as the temporal prediction model performed no better than random guessing. The difference in survival rates is attributed to the different operational frameworks: human developers actively maintain and evolve code, whereas AI-generated code often sits as a stable, albeit potentially flawed, artifact.