Agent Skills are Broken - Vercel

Quick Overview

The Vercel team discovered that standard AI agent skills fail reliably because agents often ignore available tools (skills), even when those skills are relevant, leading to poor performance; they solved this by implementing AGENTS.md, which provides persistent, version-matched context, boosting the pass rate from 53% (baseline) to 100% in evaluations.

Key Points: Standard AI agent skills failed reliably, with the skill never being invoked in 56% of evaluation cases, resulting in a 53% pass rate, identical to the baseline without any docs. Explicit instructions added to AGENTS.md improved the skill trigger rate to 95%+ and boosted the pass rate to 79% (+26 percentage points over baseline). The superior approach, using AGENTS.md as a docs index, achieved a perfect 100% pass rate (+47 percentage points over baseline). The core problem is that agents rely on outdated training data and frequently ignore available skills, which can introduce noise or distraction if unused. AGENTS.md functions as a persistent context file in the project root, available to the agent every turn without the agent needing to decide whether to load it. Contextual information (like skill definitions) was compressed from an initial 40KB injection down to 8KB (an 80% reduction) while maintaining the 100% pass rate.

Context: The video discusses an evaluation conducted by Vercel engineers regarding the reliability of AI coding agents utilizing 'skills'—packaged domain knowledge, tools, and documentation—to solve Next.js tasks. The initial approach relied on agents recognizing when to invoke these skills, but testing revealed agents often failed to use them or were confused by outdated training data, leading to incorrect code generation or reliance on older patterns.

Detailed Analysis

Vercel explored methods for teaching AI coding agents framework-specific knowledge using 'skills,' which bundle prompts, tools, and documentation that agents can invoke on demand. The initial approach, which relied on the agent recognizing when to invoke a skill, proved unreliable; in 56% of evaluation cases, the skill was never invoked, yielding a 53% pass rate, matching the baseline without any documentation. Explicit instructions added to an AGENTS.md file improved the trigger rate to over 95% and boosted the pass rate to 79% (+26 percentage points), confirming that direct instruction helps, but the wording is fragile. Further testing showed that using AGENTS.md as a comprehensive, indexed documentation source, rather than relying on skill invocation alone, achieved a perfect 100% pass rate (+47 percentage points). The video contrasts two methods: providing explicit instructions (which helped but was fragile) versus using AGENTS.md as a persistent context file containing an index of documentation, which the agent can access every turn without needing to decide to load it, yielding superior results. Furthermore, context bloat was addressed by compressing the initial 40KB injection of documentation down to 8KB (an 80% reduction) using a pipe-delimited structure, maintaining the 100% pass rate.