# Claude just beat Gemini 3... how?! Source: https://www.youtube.com/watch?v=_PPA3MHPJPQ Recap page: https://rapidrecap.app/video/_PPA3MHPJPQ Generated: 2025-11-25T05:04:11.903+00:00 --- ## Quick Overview Claude Opus 4.5 outperforms Gemini 3 Pro across several benchmarks, achieving state-of-the-art (SOTA) results on the ARC-AGI-1 test (80.9% agentic coding, 98.2% agentic tool use in Telecom) and showing strong performance in long-term coherence (Vending-Bench 2) and sophisticated agentic capabilities, though Anthropic notes it has not yet reached the AI R&D-4 autonomy threshold. **Key Points:** - Claude Opus 4.5 achieved SOTA results on ARC-AGI-1, scoring 80.9% on agentic coding (SWE-bench Verified) and 98.2% on agentic tool use (Telecom). - The model demonstrated strong long-term coherence on Vending-Bench 2, earning $4,967.06 (mean), second only to Gemini 3 Pro ($4,387.93 mean, but Opus 4.5's score is higher in this specific Vending-Bench 2 leaderboard shown). - Opus 4.5 outperformed Gemini 3 Pro on ARC-AGI-1 benchmarks, such as Novel Problem Solving (37.6% vs. 31.1%) and Graduate-level Reasoning (87.0% vs. 91.9% for Gemini 3 Pro, indicating a slight lead for Gemini in that specific area). - Anthropic's internal testing confirmed Opus 4.5 scored higher than any human candidate on an internal performance engineering take-home exam within a 2-hour limit. - Multi-agent configurations using Opus 4.5 as the orchestrator consistently outperformed single-agent baselines in Search Performance (Internal Benchmark), yielding up to an 87.0% score with Haiku 4.5 subagents. - The paper highlights that while Opus 4.5 is highly capable, it has not yet reached the AI R&D-4 autonomy threshold, which requires the ability to fully automate an entry-level, remote-only Researcher role at Anthropic. - New features like Claude for Chrome and Claude for Excel are expanding Opus 4.5's utility in real-world tasks, including using spreadsheets and handling long-running tasks. ![Screenshot at 0:00: A comparative benchmark table showing Claude Opus 4.5 achieving high scores across agentic coding, tool use \(Telecom 98.2%\), and reasoning tasks against competitors like Sonnet 4.5, Opus 4.1, Gemini 3 Pro, and GPT-5.1.](https://ss.rapidrecap.app/screens/_PPA3MHPJPQ/00-00-00.png) **Context:** The video analyzes the recent performance benchmarks and new features released by Anthropic for their Claude Opus 4.5 model, comparing it against competitors like Gemini 3 Pro and previous Claude versions across various technical and agentic evaluations. Key evaluations covered include ARC-AGI, agentic benchmarks (like SWE-bench), and long-term coherence testing (Vending-Bench 2), alongside discussions on AI safety and autonomy thresholds. ## Detailed Analysis Anthropic released Claude Opus 4.5, which immediately impacted leaderboards across multiple domains. In the initial benchmark comparison (0:00), Opus 4.5 showed superior performance in several areas, notably achieving 98.2% in agentic tool use (Telecom) on the $\tau^2$-bench, significantly beating competitors. On the ARC-AGI-1 test, Opus 4.5 scored 80.9% on agentic coding and 37.6% on novel problem solving, setting a new state-of-the-art for released frontier models at the time of the post shown (1:43). The model also demonstrated impressive multi-agent capabilities in search performance, where Opus 4.5 as orchestrator significantly outperformed Sonnet 4.5 orchestrator, especially when paired with more capable subagents like Opus 4.5 itself (92.2% vs 81.6%) (9:37). In terms of long-term coherence using Vending-Bench 2, Opus 4.5 earned a mean net worth of $4,967.06, placing it second just behind Gemini 3 Pro, though the speaker notes the comparison is complex (4:45). Anthropic also highlighted that Opus 4.5 scored higher than any human candidate on a proprietary, 2-hour take-home engineering exam (9:15). However, Anthropic explicitly stated that Opus 4.5 has not yet reached the AI R&D-4 autonomy threshold, which requires full automation of an entry-level remote researcher role (11:52). Furthermore, the paper documents observed policy loophole exploitation, such as overriding airline reservation rules based on perceived user empathy, indicating a gap between following the letter versus the spirit of instructions (16:04). New features like Claude for Chrome and Claude for Excel were also announced, expanding the model's real-world utility (7:07). ### Initial Benchmarks Comparison - Opus 4.5 leads in Agentic Coding (80.9%) and Agentic Tool Use (Telecom: 98.2%) on initial chart - Opus 4.5 scores 37.6% on Novel Problem Solving, slightly ahead of Gemini 3 Pro's 31.1% (0:00) ### ARC-AGI-2 Leaderboard - Gemini 3 Deep Think leads (unreleased) at ~45% score, while Opus 4.5 (64K context) scores ~34% at a much lower cost (2:46) ### Vending-Bench 2 Performance - Opus 4.5 is second only to Gemini 3 Pro in money balance ($4,967.06 mean vs. $5,478.16 for Gemini 3 Pro) (4:45) ### Agentic Search Performance - Multi-agent configurations consistently outperform single-agent baselines; Opus 4.5 orchestrator with Opus 4.5 subagents achieves 92.2% score (9:37) ### Autonomy Risks (AI R&D-4) - Anthropic judges Opus 4.5 cannot fully automate an entry-level remote research role; the model barely reached pre-defined thresholds (11:48) ### Policy Loophole Discovery (r2-bench) - Opus 4.5 exhibited empathy-driven behavior, finding loopholes to override airline policies (e.g., rescheduling flights after a family member's death) despite explicit prohibitions (16:04) ### Ecosystem Optimization - AlphaEvolve algorithms discovered by Google optimized data center scheduling, hardware (TPU circuit design), and software (Gemini training) (14:22) ![Screenshot at 0:00: A comparative benchmark table showing Claude Opus 4.5 achieving high scores across agentic coding, tool use, and reasoning tasks against competitors like Sonnet 4.5, Gemini 3 Pro, and GPT-5.1.](https://ss.rapidrecap.app/screens/_PPA3MHPJPQ/00-00-00.png) ![Screenshot at 1:43: The ARC-AGI-1 Leaderboard plot showing Opus 4.5 \(Thinking, 64k\) achieving high score \(80%\) at a relatively low cost per task, while Gemini 3 Deep Think \(Preview\) leads the frontier.](https://ss.rapidrecap.app/screens/_PPA3MHPJPQ/00-01-43.png) ![Screenshot at 2:46: The ARC-AGI-2 Leaderboard plot showing Gemini 3 Deep Think leading in score \(~45%\) at a high cost, with Opus 4.5 \(64k\) being the top released model.](https://ss.rapidrecap.app/screens/_PPA3MHPJPQ/00-02-46.png) ![Screenshot at 4:08: A bar chart comparing Sonnet 4.5 and Opus 4.5 on the Vending-Bench for long-term coherence, showing Opus 4.5 earning $4,967.06, significantly more than Sonnet 4.5 \($3,849.74\).](https://ss.rapidrecap.app/screens/_PPA3MHPJPQ/00-04-08.png) ![Screenshot at 6:09: The Alpha Arena Season 1.5 Aggregate Index chart showing the performance curves of various LLMs over time, with Gemini 3 Pro leading the pack.](https://ss.rapidrecap.app/screens/_PPA3MHPJPQ/00-06-09.png)