# Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

Source: https://www.youtube.com/watch?v=YKLnwv7AyX8
Recap page: https://rapidrecap.app/video/YKLnwv7AyX8
Generated: 2026-01-27T15:46:02.55+00:00

---
## Quick Overview

The Terminal Bench benchmark, created by researchers from institutions like Stanford and Anthropic, reveals that even frontier models like GPT 5.2 fail on hard, realistic command line interface tasks more than 35% of the time, highlighting a critical gap between theoretical coding ability and reliable operational execution.

**Key Points:**
- Frontier models, including GPT 5.2 and Claude Opus 4.5, still fail more than 35% of the time when placed in a realistic, unguided terminal environment.
- The best performer, Codeex CLI paired with GPT 5.2, solved 63% of the 89 outcome-driven tasks within the Terminal Bench suite.
- The single biggest cause of failure, accounting for over 24% of errors, is 'command not found,' indicating agents hallucinate tools that are not installed in the environment.
- The benchmark isolates raw model intelligence by using Terminus 2, a minimalist scaffold that pipes raw output directly into a bash shell without safety nets.
- A significant failure mode is 'reasoning action mismatch,' where logs show the agent reasoning that it completed a check (like running 'ls') when the command was never actually executed.
- The benchmark includes diverse, high-stakes tasks such as rewriting Cobalt programs to be byte-for-byte identical in Python and configuring complex legacy systems like installing Windows 3.11 within an emulator.
- There is zero correlation between the number of turns an agent takes and success; GPT5 Nano generated the most tokens (60,000 average) but achieved only an 8% success rate.

**Context:** The research introduces Terminal Bench, a new benchmark designed to test AI agents on complex, realistic tasks within command line interfaces (CLIs), which researchers call the 'cockpit of modern computing' and the 'nervous system' of a machine due to its direct, unfiltered access. This work moves beyond traditional text-in, text-out chatbot evaluations to test agent systems that must actually execute actions, involving authors from Stanford, Anthropic, and MIT.

## Detailed Analysis

Terminal Bench shifts the focus from simple text generation to agent execution in the command line, presenting 89 outcome-driven tasks that mimic what a human power user performs, ranging from Cobol modernization requiring byte-for-byte replication in Python to complex DevOps tasks like setting up QEMU and a web server to run Windows 3.11. The evaluation framework uses Terminus 2, a deliberately minimalist agent scaffold to isolate the model's raw intelligence by feeding output directly into a bash shell without product-level safety nets. Results show that even leading models struggle significantly; GPT 5.2 achieved 63% success, while Claude Opus 4.5 reached 58%, confirming that reasoning in a theoretical space differs vastly from live execution. Error analysis points to fundamental flaws: over 24% of failures stem from 'command not found' hallucinations, and a worrying 'reasoning action mismatch' occurs when agents assume they executed a command when they did not. Despite these failures, performance is accelerating rapidly, nearly doubling between Gemini 2.5 Pro and GPT 5.2 in eight months, suggesting the benchmark may saturate soon, but the current reliability issues mandate continued mandatory supervision for production deployment.

### Benchmark Purpose and Design

- Moving beyond chatbots to test agents on action execution in the CLI
- The benchmark covers 89 outcome-driven tasks
- Verification included an adversarial exploit agent and three hours of human auditing per task

### Performance Metrics and Leaderboard

- GPT 5.2 paired with Codeex CLI achieved the state-of-the-art at 63% success
- Open-weight models like Kei K2 thinking lagged significantly at 36% success

### Task Diversity and Difficulty

- Tasks included rewriting legacy Cobol identically in Python and setting up complex virtual environments for old operating systems
- A multimodal task required interpreting a chessboard image to find the best move in algebraic notation

### Failure Analysis

- The primary failure mode is 'command not found' (24% of failures), showing agents hallucinate available tools
- 'Reasoning action mismatch' is scary, where agents assert they checked for a file but never ran the 'ls' command

### Model Behavior Insights

- There is zero connection between effort (number of turns/tokens generated) and success
- GPT5 Nano generated the most tokens but succeeded only 8% of the time, contrasting with efficient master carpenters

### Research Limitations and Future Pace

- The benchmark is fragile due to reliance on external package repositories (the 'internet problem')
- A creative adversarial reasoning gap exists, where models struggle with tasks humans find easy, like bypassing a web filter

### Implications for Deployment

- Progress velocity is stunning, with performance nearly doubling in eight months
- The high failure rate, especially basic execution errors, serves as a huge warning label against granting unsupervised right access to production servers