Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

Quick Overview

The Terminal Bench benchmark, created by researchers from institutions like Stanford and Anthropic, reveals that even frontier models like GPT 5.2 fail on hard, realistic command line interface tasks more than 35% of the time, highlighting a critical gap between theoretical coding ability and reliable operational execution.

Key Points: Frontier models, including GPT 5.2 and Claude Opus 4.5, still fail more than 35% of the time when placed in a realistic, unguided terminal environment. The best performer, Codeex CLI paired with GPT 5.2, solved 63% of the 89 outcome-driven tasks within the Terminal Bench suite. The single biggest cause of failure, accounting for over 24% of errors, is 'command not found,' indicating agents hallucinate tools that are not installed in the environment. The benchmark isolates raw model intelligence by using Terminus 2, a minimalist scaffold that pipes raw output directly into a bash shell without safety nets. A significant failure mode is 'reasoning action mismatch,' where logs show the agent reasoning that it completed a check (like running 'ls') when the command was never actually executed. The benchmark includes diverse, high-stakes tasks such as rewriting Cobalt programs to be byte-for-byte identical in Python and configuring complex legacy systems like installing Windows 3.11 within an emulator. There is zero correlation between the number of turns an agent takes and success; GPT5 Nano generated the most tokens (60,000 average) but achieved only an 8% success rate.

Context: The research introduces Terminal Bench, a new benchmark designed to test AI agents on complex, realistic tasks within command line interfaces (CLIs), which researchers call the 'cockpit of modern computing' and the 'nervous system' of a machine due to its direct, unfiltered access. This work moves beyond traditional text-in, text-out chatbot evaluations to test agent systems that must actually execute actions, involving authors from Stanford, Anthropic, and MIT.