# Anthropic: Quantifying Infrastructure Noise in Agentic Coding Evals Source: https://www.youtube.com/watch?v=O0rWHOxnF7E Recap page: https://rapidrecap.app/video/O0rWHOxnF7E Generated: 2026-02-07T12:03:45.589+00:00 --- ## Quick Overview The Anthropic research found that infrastructure noise, specifically related to infrastructure setup like CPU and RAM limits, significantly impacts the performance scores of agentic coding evaluations, causing failures not due to model intelligence but environment constraints, leading to a recommendation to standardize infrastructure or account for this noise in evaluations. **Key Points:** - The paper quantifies infrastructure noise's effect on agentic coding evaluations, finding it causes failure spikes. - A 1x setup (4 CPUs, 16GB RAM) with a 20GB hard limit failed 5.8% of tasks due to infrastructure errors. - The same model on a 5x setup (20 CPUs, 80GB RAM, 20GB limit) failed only 2.1% of tasks, demonstrating the impact of resource allocation. - The authors suggest that infrastructure noise causes a 6 percentage point difference between the strictest and loosest test configurations. - Infrastructure noise is deemed an active part of the problem because it can cause catastrophic failures (like an oven randomly shutting off during baking) that mask underlying model capability. - The solution proposed is to either standardize the infrastructure for all tests or explicitly account for infrastructure noise when reporting scores, as the current process often ignores this factor. - The benchmark 'BinCh 2.0' was used to compare the strict (1x) setup against the looser (5x) setup. ![Screenshot at 0:09: The introduction explicitly frames the discussion around how infrastructure noise challenges the perception of the whole AI landscape, setting up the central problem of the paper.](https://ss.rapidrecap.app/screens/O0rWHOxnF7E/00-00-09.jpg) **Context:** The video discusses research from Anthropic titled "Quantifying Infrastructure Noise in Agentic Coding Evals," which investigates how the computational environment—specifically CPU, RAM, and disk space limits—affects the reported performance scores of AI agents during coding evaluations. The core issue is that infrastructure limitations can cause task failures that are incorrectly attributed to the model's intelligence rather than environmental constraints, creating a misleading leaderboard. ## Detailed Analysis The Anthropic research focuses on 'Quantifying Infrastructure Noise in Agentic Coding Evals,' arguing that infrastructure setup heavily influences performance scores, often masking true model capabilities. The researchers quantified this by comparing two setups: a strict 1x setup (4 CPUs, 16GB RAM, 20GB limit) and a looser 5x setup (20 CPUs, 80GB RAM, 20GB limit). On the strict 1x setup, 5.8% of tasks failed due to infrastructure errors, whereas the 5x setup only failed 2.1% of the time, showing a 6 percentage point gap caused by resource constraints. The paper draws an analogy to testing a carpenter: giving them a tiny workshop versus a large warehouse to build a souffle—the environment dictates the outcome, not just the carpenter's skill. The researchers call this effect 'infrastructure noise' and note that it leads to failures that are not due to the model being 'dumb' but due to environmental issues like container crashes or running out of memory during library loading. The key takeaway is that leaderboard scores that don't account for infrastructure noise are misleading, as they reward models that run well on the specific (and often resource-rich) testing environment, rather than models with better inherent capabilities. The paper recommends either standardizing the infrastructure across all evaluations or explicitly documenting and accounting for the noise, suggesting that a simple metric like the number of tasks that fail due to infrastructure errors is critical context for any reported score. ### AI Arms Race Context - Companies fighting publicly on leaderboards - Perception of razor-thin competition - Decisions involving millions of dollars depend on these scores ### The Infrastructure Noise Experiment - Paper titled 'Quantifying Infrastructure Noise in Agentic Coding Evals' - Compared strict 1x setup vs. loose setup - Strict setup failed 5.8% due to noise, loose setup failed 2.1% on the same tasks ### Phase 1 - Stabilization: Moving from a failing environment to a stable one - Getting rid of bad luck - The goal is to create a level playing field ### Phase 2 - Capability Unlocking: Moving from strict (1x) to loose (5x) resources - Model can try different strategies - The test should be for model capability, not resourcefulness ### Conclusion and Takeaway - Infrastructure noise is an active problem that blurs the line between capability and luck - The 1x setup's error rate (2.1%) is misleadingly high compared to the 5x setup's true error rate - Documenting infrastructure is crucial for fair evaluation ![Screenshot at 0:09: The visual shows the podcast hosts against a grid background with an audio wave, emphasizing the focus on current, measurable data in AI research.](https://ss.rapidrecap.app/screens/O0rWHOxnF7E/00-00-09.jpg) ![Screenshot at 0:36: A visual representation of the leaderboards being treated as 'perfectly calibrated scientific instruments,' which the speaker argues is an illusion due to infrastructure noise.](https://ss.rapidrecap.app/screens/O0rWHOxnF7E/00-00-36.jpg) ![Screenshot at 1:55: The speaker introduces the paper's core finding, which makes a sharp distinction between static benchmarks and the reality of agentic evaluation.](https://ss.rapidrecap.app/screens/O0rWHOxnF7E/00-01-55.jpg) ![Screenshot at 4:37: The speaker mentions the hidden variable: the time of day factor affecting test results, which is another element of environmental noise.](https://ss.rapidrecap.app/screens/O0rWHOxnF7E/00-04-37.jpg) ![Screenshot at 12:22: A slide or graphic summarizing the two phases of improvement discovered in the research: Stabilization \(Phase 1\) and Capability Unlocking \(Phase 2\).](https://ss.rapidrecap.app/screens/O0rWHOxnF7E/00-12-22.jpg)