# OpenAI - EVMbench: Evaluating AI Agents on Smart Contract Security Source: https://www.youtube.com/watch?v=ETZbSD4mu2U Recap page: https://rapidrecap.app/video/ETZbSD4mu2U Generated: 2026-02-21T23:05:24.811+00:00 --- ## Quick Overview The EVMbench paper evaluates AI agents' ability to secure smart contracts by testing their performance in three modes: Detect, Patch, and Exploit, finding that while agents are excellent at detecting known vulnerabilities (72.2% success rate), they struggle with exploiting complex, novel vulnerabilities and performing end-to-end exploitation, indicating that human security expertise remains crucial in the current crypto landscape. **Key Points:** - The EVMbench framework tests AI agents on smart contract security tasks across three modes: Detect, Patch, and Exploit. - In Detect mode, agents achieved a high success rate of 72.2% in finding known vulnerabilities, outperforming humans (45% success rate in finding known issues). - The agents demonstrated difficulty in exploiting vulnerabilities, particularly those requiring complex timing or historical context, like the flash loan attack simulation. - The primary bottleneck identified is the search problem—finding the specific exploit code—rather than the technical coding skill, as agents wrote correct code when guided. - The EVMbench environment is an isolated, deterministic simulation of the Ethereum mainnet, allowing for repeatable testing without real financial risk. - Agents struggled with issues requiring semantic understanding, like the blank slate issue in the EVM, suggesting a gap between syntactic correctness and economic capability. ![Screenshot at 00:00: The video opens with the 'Really Easy AI' podcast branding, featuring an image of two hosts at microphones over a grid, overlaid with the call to action 'BECOME A MEMBER TODAY!', setting the stage for a discussion about a technical paper.](https://ss.rapidrecap.app/screens/ETZbSD4mu2U/00-00-00.jpg) **Context:** The video discusses the EVMbench paper, which introduces a new benchmark designed to rigorously evaluate the security auditing capabilities of AI agents on Ethereum Virtual Machine (EVM) smart contracts. This evaluation moves beyond simple code generation to test real-world security tasks like finding, patching, and exploiting vulnerabilities, specifically focusing on whether current AI agents can function as effective security auditors or attackers in the volatile blockchain environment. ## Detailed Analysis The EVMbench paper introduces a structured evaluation method for AI agents assessing smart contract security, using three modes: Detect, Patch, and Exploit. The agents were tested against 120 specific vulnerabilities sourced from real-world Code4rena audit contests. In the Detect mode, agents performed strongly, successfully identifying known vulnerabilities 72.2% of the time, significantly higher than the 45% rate observed for human auditors in similar scenarios. However, this high detection rate is misleading because the agents struggled significantly with the subsequent exploitation mode, where they only succeeded 52.2% of the time. The paper highlights that the main difficulty is not writing the exploit code itself (which agents can do well when guided), but rather finding the exploit within the massive state space of the blockchain (the search problem). The EVMbench simulation environment mirrors the Ethereum mainnet exactly, allowing for reproducible testing. The agents also failed to exploit vulnerabilities that required complex, temporally sensitive actions, such as flash loan attacks or exploiting logic flaws that were not immediately apparent. The conclusion is that while AI excels at finding known issues, the complexity of real-world exploitation and nuanced economic reasoning (like identifying subtle economic exploits) still requires significant human oversight, suggesting AI is currently better suited as a security assistant than a fully autonomous auditor or attacker. ### EVMbench Overview - Defines the benchmark for evaluating AI agents on smart contract security - uses three modes: Detect, Patch, Exploit - employs 120 real-world vulnerabilities from Code4rena contests. ### Detection Performance - AI agents achieved a 72.2% success rate in detecting known vulnerabilities, outperforming human auditors' 45% rate on similar issues. ### Exploitation Performance - Agent success rate dropped significantly in Exploit mode to 52.2%, indicating difficulty in executing complex attacks. ### Bottleneck Identification - The primary limitation is the search space (finding the exploit) rather than coding ability, as agents successfully wrote code when given hints. ### Key Vulnerability Types - Agents struggled with vulnerabilities requiring complex timing or historical context, like the flash loan attack simulation. ### Comparison to Human Auditors - Agents are better at finding known bugs but struggle more than humans when dealing with complex, novel exploits or economic reasoning flaws. ![Screenshot at 00:00: The opening screen displays the podcast branding and a call to action to 'BECOME A MEMBER TODAY!', signaling the start of the discussion.](https://ss.rapidrecap.app/screens/ETZbSD4mu2U/00-00-00.jpg) ![Screenshot at 00:15: The speaker introduces the paper's focus: 'EVMBench Evaluating AI Agents on Smart Contract Security', setting the central topic.](https://ss.rapidrecap.app/screens/ETZbSD4mu2U/00-00-15.jpg) ![Screenshot at 02:10: The speaker details the scope of the vulnerabilities tested, mentioning they curated 120 specific vulnerabilities from real-world sources.](https://ss.rapidrecap.app/screens/ETZbSD4mu2U/00-02-10.jpg) ![Screenshot at 03:35: The speaker outlines the three evaluation modes: Detect, Patch, and Exploit, which form the core testing methodology.](https://ss.rapidrecap.app/screens/ETZbSD4mu2U/00-03-35.jpg) ![Screenshot at 07:52: A statistic appears showing the AI's success rate in exploitation mode \(72.2%\), highlighting the performance difference between detection and exploitation.](https://ss.rapidrecap.app/screens/ETZbSD4mu2U/00-07-52.jpg)