# Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? Source: https://www.youtube.com/watch?v=tNnxDpmvBKE Recap page: https://rapidrecap.app/video/tNnxDpmvBKE Generated: 2026-02-28T20:03:25.753+00:00 --- ## Quick Overview The research demonstrated that including repository-level context files in prompts for coding agents, such as those used by agents like GPT-4.5 or Anthropic's models, significantly degrades performance, causing agents to spend more time on irrelevant tasks and resulting in lower success rates compared to using only task-specific instructions. **Key Points:** - Including repository-level context files in prompts for coding agents significantly degrades performance, leading to higher costs and lower success rates. - The study found that using a context file led to a 22% higher reasoning token count compared to not using one when testing agents like GPT-4.5 and Claude 3. - Agents using the context file spent 22% more time on reasoning and were prone to following the context file's structure too closely, even when it was irrelevant or noisy. - The AgentBench evaluation showed that when provided with the context file, success rates dropped by 0.5% to 3% across models, indicating that the context file actively hurt performance. - The context file often contained noise or was overly long (e.g., a 5 million line repo for a small task), causing agents to over-rely on it instead of focusing on the immediate task. - The authors recommend that developers avoid including generic architectural overviews or file trees in prompts, favoring strictly minimal, human-written instructions for better outcomes. ![Screenshot at 00:14: The speaker explicitly states that the inclusion of context files causes the agent's performance to drop, highlighting the core negative finding of the research being presented.](https://ss.rapidrecap.app/screens/tNnxDpmvBKE/00-00-14.jpg) **Context:** This video discusses the findings of a research paper evaluating the impact of providing Large Language Models (LLMs), specifically coding agents, with extensive context about the entire repository structure via a context file (agents.md). The context file, which usually contains high-level architecture or file structure information, was tested against models like GPT-4.5 and Claude 3 to see if it helped them write better code or debug existing code, particularly when compared to agents relying only on minimal, human-written instructions. ## Detailed Analysis The research investigated whether providing coding agents with a repository-level context file, exemplified by an agents.md file, actually helps or hurts performance. The study found that providing this context file is counterproductive, acting as a distraction that leads to higher computational costs and lower success rates. When agents like GPT-4.5 and Claude 3 were given the context file, they spent 22% more reasoning tokens and experienced a performance drop of 0.5% to 3% on average compared to when they only received specific instructions. The agents tended to follow the structure laid out in the context file too closely, even when it meant ignoring the actual debugging task or writing code that didn't exist in the provided structure. The researchers specifically tested scenarios where agents were given either the context file or a completely empty file (a 'ghost town' scenario) and found that the context file often led to worse results, suggesting it provided noise rather than helpful information. The ultimate takeaway is that for developers, giving agents minimal, highly specific, human-written instructions is far more effective than providing a broad, potentially noisy map of the entire code base. ### Introduction to Context File Testing - Agents are usually given context files like agents.md or similar high-level overviews - The experiment tests if this context helps agents navigate complex codebases or if it leads to over-reliance and errors. ### Experimental Results - Performance Drop - Agents using the context file showed a 22% increase in reasoning tokens and a 0.5% to 3% drop in success rate compared to agents without it. - The cost of running the agent increased by over 20% due to the extra processing. ### Agent Behavior Analysis - Agents followed the context file structure too closely, even when it was irrelevant or contained noise, failing to prioritize the specific task at hand. - This led to agents trying to fix non-existent issues or ignoring the actual bug fix needed. ### Comparison of Model Behavior - GPT-4.5 and Claude 3, despite being advanced models, showed similar negative tendencies when provided with the context file. ### Conclusion and Recommendation - The study concludes that providing extensive context files is counterproductive; developers should stick to strictly minimal, human-written instructions that focus only on the immediate task. ![Screenshot at 00:00: The video opens with the podcast graphic urging viewers to 'Become a member today!' over an audio waveform visualization.](https://ss.rapidrecap.app/screens/tNnxDpmvBKE/00-00-00.jpg) ![Screenshot at 00:25: The speaker introduces the context of the discussion, mentioning developers working with AI agents and the necessity of context files like agents.md.](https://ss.rapidrecap.app/screens/tNnxDpmvBKE/00-00-25.jpg) ![Screenshot at 00:56: The speaker describes the issue, stating that a model's logic seems 'ironclad on the surface' but that giving it a cheat sheet can be counterproductive.](https://ss.rapidrecap.app/screens/tNnxDpmvBKE/00-00-56.jpg) ![Screenshot at 02:27: The speaker names the specific benchmark used in the study, 'Agent Bench,' and mentions the context file structure being evaluated.](https://ss.rapidrecap.app/screens/tNnxDpmvBKE/00-02-27.jpg) ![Screenshot at 03:36: The speaker highlights the counterintuitive finding that explicit guidance, while seemingly helpful, can hurt performance, comparing it to a programmer getting lost in documentation.](https://ss.rapidrecap.app/screens/tNnxDpmvBKE/00-03-36.jpg)