# Agent Skills are Broken - Vercel

Source: https://www.youtube.com/watch?v=A_1ELXEVp5w
Recap page: https://rapidrecap.app/video/A_1ELXEVp5w
Generated: 2026-01-30T15:02:04.95+00:00

---
## Quick Overview

The Vercel team discovered that standard AI agent skills fail reliably because agents often ignore available tools (skills), even when those skills are relevant, leading to poor performance; they solved this by implementing AGENTS.md, which provides persistent, version-matched context, boosting the pass rate from 53% (baseline) to 100% in evaluations.

**Key Points:**
- Standard AI agent skills failed reliably, with the skill never being invoked in 56% of evaluation cases, resulting in a 53% pass rate, identical to the baseline without any docs.
- Explicit instructions added to AGENTS.md improved the skill trigger rate to 95%+ and boosted the pass rate to 79% (+26 percentage points over baseline).
- The superior approach, using AGENTS.md as a docs index, achieved a perfect 100% pass rate (+47 percentage points over baseline).
- The core problem is that agents rely on outdated training data and frequently ignore available skills, which can introduce noise or distraction if unused.
- AGENTS.md functions as a persistent context file in the project root, available to the agent every turn without the agent needing to decide whether to load it.
- Contextual information (like skill definitions) was compressed from an initial 40KB injection down to 8KB (an 80% reduction) while maintaining the 100% pass rate.

![Screenshot at 0:11: Diagram illustrating the architecture of Agent + Skills + Virtual Machine, showing Agent Configuration feeding into the Agent Virtual Machine's File System, which houses the skills accessible by Bash, Python, or Node.js environments.](https://ss.rapidrecap.app/screens/A_1ELXEVp5w/00-00-11.jpg)

**Context:** The video discusses an evaluation conducted by Vercel engineers regarding the reliability of AI coding agents utilizing 'skills'—packaged domain knowledge, tools, and documentation—to solve Next.js tasks. The initial approach relied on agents recognizing when to invoke these skills, but testing revealed agents often failed to use them or were confused by outdated training data, leading to incorrect code generation or reliance on older patterns.

## Detailed Analysis

Vercel explored methods for teaching AI coding agents framework-specific knowledge using 'skills,' which bundle prompts, tools, and documentation that agents can invoke on demand. The initial approach, which relied on the agent recognizing when to invoke a skill, proved unreliable; in 56% of evaluation cases, the skill was never invoked, yielding a 53% pass rate, matching the baseline without any documentation. Explicit instructions added to an AGENTS.md file improved the trigger rate to over 95% and boosted the pass rate to 79% (+26 percentage points), confirming that direct instruction helps, but the wording is fragile. Further testing showed that using AGENTS.md as a comprehensive, indexed documentation source, rather than relying on skill invocation alone, achieved a perfect 100% pass rate (+47 percentage points). The video contrasts two methods: providing explicit instructions (which helped but was fragile) versus using AGENTS.md as a persistent context file containing an index of documentation, which the agent can access every turn without needing to decide to load it, yielding superior results. Furthermore, context bloat was addressed by compressing the initial 40KB injection of documentation down to 8KB (an 80% reduction) using a pipe-delimited structure, maintaining the 100% pass rate.

### Agent Skills Introduction

- Skills are an open standard for packaging domain knowledge; AGENTS.md is a markdown file providing persistent context to coding agents.

### Initial Skill Evaluation Failure

- Skills weren't triggered reliably; in 56% of eval cases, the skill was never invoked, resulting in a 53% pass rate, suggesting unused skills introduce noise.

### Approach 1

- Explicit Instructions: Adding instructions like 'Before writing code, first explore the project structure, then invoke the nextjs-doc skill' improved the pass rate to 79% (+26pp).

### Approach 2

- AGENTS.md Docs Index: Stuffing all context into AGENTS.md as a summarized, indexed version of available skills yielded a 100% pass rate (+47pp over baseline).

### Addressing Context Bloat

- Initial 40KB context injection was compressed to 8KB (80% reduction) using a pipe-delimited structure, maintaining the 100% pass rate.

![Screenshot at 0:11: Diagram illustrating the Agent + Skills + Virtual Machine architecture, detailing the configuration and file system interaction.](https://ss.rapidrecap.app/screens/A_1ELXEVp5w/00-00-11.jpg)
![Screenshot at 0:21: Example of a simple SKILL.md file structure showing YAML Frontmatter \(name, description\) and Markdown content \(Overview, Quick Start\).](https://ss.rapidrecap.app/screens/A_1ELXEVp5w/00-00-21.jpg)
![Screenshot at 0:49: Context Usage breakdown for Claude-Sonnet-4 showing that 'MCP tools' consume 44.6% of the context window \(89.1k tokens\).](https://ss.rapidrecap.app/screens/A_1ELXEVp5w/00-00-49.jpg)
![Screenshot at 1:03: Diagram showing how 'Skills and the Context Window' work: short skill snippets are appended to the system prompt, followed by the user message, allowing Claude to decide to trigger a skill via tool use.](https://ss.rapidrecap.app/screens/A_1ELXEVp5w/00-01-03.jpg)
![Screenshot at 1:24: Evaluation table comparing configurations: Baseline \(no docs\) 53% pass rate vs. Skill \(default behavior\) 53% pass rate, showing no initial benefit from skills.](https://ss.rapidrecap.app/screens/A_1ELXEVp5w/00-01-24.jpg)