Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Quick Overview

The research demonstrated that including repository-level context files in prompts for coding agents, such as those used by agents like GPT-4.5 or Anthropic's models, significantly degrades performance, causing agents to spend more time on irrelevant tasks and resulting in lower success rates compared to using only task-specific instructions.

Key Points: Including repository-level context files in prompts for coding agents significantly degrades performance, leading to higher costs and lower success rates. The study found that using a context file led to a 22% higher reasoning token count compared to not using one when testing agents like GPT-4.5 and Claude 3. Agents using the context file spent 22% more time on reasoning and were prone to following the context file's structure too closely, even when it was irrelevant or noisy. The AgentBench evaluation showed that when provided with the context file, success rates dropped by 0.5% to 3% across models, indicating that the context file actively hurt performance. The context file often contained noise or was overly long (e.g., a 5 million line repo for a small task), causing agents to over-rely on it instead of focusing on the immediate task. The authors recommend that developers avoid including generic architectural overviews or file trees in prompts, favoring strictly minimal, human-written instructions for better outcomes.

Context: This video discusses the findings of a research paper evaluating the impact of providing Large Language Models (LLMs), specifically coding agents, with extensive context about the entire repository structure via a context file (agents.md). The context file, which usually contains high-level architecture or file structure information, was tested against models like GPT-4.5 and Claude 3 to see if it helped them write better code or debug existing code, particularly when compared to agents relying only on minimal, human-written instructions.

Detailed Analysis

The research investigated whether providing coding agents with a repository-level context file, exemplified by an agents.md file, actually helps or hurts performance. The study found that providing this context file is counterproductive, acting as a distraction that leads to higher computational costs and lower success rates. When agents like GPT-4.5 and Claude 3 were given the context file, they spent 22% more reasoning tokens and experienced a performance drop of 0.5% to 3% on average compared to when they only received specific instructions. The agents tended to follow the structure laid out in the context file too closely, even when it meant ignoring the actual debugging task or writing code that didn't exist in the provided structure. The researchers specifically tested scenarios where agents were given either the context file or a completely empty file (a 'ghost town' scenario) and found that the context file often led to worse results, suggesting it provided noise rather than helpful information. The ultimate takeaway is that for developers, giving agents minimal, highly specific, human-written instructions is far more effective than providing a broad, potentially noisy map of the entire code base.