# Buckle Up, Buttercup - Our Experience Competing in AIxCC Source: https://www.youtube.com/watch?v=dVnWSQDpFhw Recap page: https://rapidrecap.app/video/dVnWSQDpFhw Generated: 2025-09-26T15:40:02.5+00:00 --- ## Quick Overview Trail of Bits' AI system, Buttercup, secured second place in the AIxCC competition, winning $3 million by effectively integrating conventional software analysis with AI/ML techniques to discover and patch vulnerabilities in open-source software, achieving over 90% accuracy and demonstrating a cost-effective approach to automated cybersecurity. **Key Points:** - Buttercup achieved second place in the AIxCC finals, securing a $3 million prize by effectively finding and patching vulnerabilities in open-source software. - The system's success was attributed to its high accuracy rate of over 90% and its ability to perform well across all competition phases, particularly in patching vulnerabilities which contributed significantly to scoring. - Buttercup utilized a hybrid approach, combining static analysis, dynamic analysis, and large language models (LLMs) to discover vulnerabilities, and employed a multi-agent system for patch generation and validation. - During the semi-finals, Trail of Bits won 10 out of 42 achievements, demonstrating strong performance, but identified gaps in handling Java challenges and certain CWE types that were addressed for the finals. - The finals presented a significant scaling challenge with vastly increased budgets for compute and LLM credits compared to the semi-finals, requiring a rebuild of Buttercup into version 2.0. - Buttercup's design prioritized using the best technique for each sub-problem, acknowledging LLM limitations and strengths, and focusing on discovering/proving vulnerabilities, contextualizing them, creating/validating patches, and orchestrating these tasks. - Trail of Bits has open-sourced both the semi-final and final versions of Buttercup, as well as a scaled-down standalone variant for laptop use, to promote the advancement of cybersecurity tools. **Context:** The AI Cyber Challenge (AIxCC) was a two-year competition where teams designed AI-driven cyber reasoning systems to automatically find and patch bugs in open-source software. Trail of Bits, represented by the team discussing their system 'Buttercup,' participated in this challenge. They began by winning a white paper contest, advancing through semi-finals held at Defcon, and ultimately competing in the finals where Buttercup secured second place and a $3 million prize. ## Detailed Analysis Trail of Bits' AI system, Buttercup, achieved second place in the AIxCC competition, earning a $3 million prize. The system's success stemmed from its hybrid approach, combining traditional software analysis techniques with AI and machine learning to automatically discover and patch vulnerabilities in open-source software. The team detailed their journey through the competition, from the semi-finals where they won numerous achievements, to the finals which required a complete rebuild of Buttercup (version 2.0) to handle significantly increased scale and complexity. Key to their strategy was breaking down the problem into distinct phases: vulnerability discovery (using fuzzing and LLM input generation), contextualization (extracting code facts), patch generation and validation (via a multi-agent system), and orchestration. The patcher, a multi-agent system using GPT-4, was particularly robust, employing security, software, and quality engineer agents, with a reflection agent to handle failures. In the finals, Buttercup demonstrated high efficiency, using only $40,000 of a large budget and achieving over 90% accuracy, which was crucial for their second-place finish over teams with lower accuracy or less successful patching rates. The team highlighted that they found at least one vulnerability no other team discovered and potentially a non-synthetic vulnerability, proving Buttercup's capability on real-world systems. Trail of Bits has open-sourced Buttercup, including a standalone version runnable on laptops, to contribute to the broader cybersecurity community. ### Competition Overview - AIxCC is a two-year competition tasking teams with designing AI-driven systems to automatically find and patch software bugs - Trail of Bits' 'Buttercup' system competed, advancing from semi-finals to finals - Finals involved finding and patching vulnerabilities within challenge projects for points and prizes. ### Buttercup System Design - Hybrid approach combining static/dynamic analysis with LLMs for vulnerability discovery - Contextualization phase extracts code facts using tree-sitter and code-query - Patch generation uses a multi-agent system (security, software, quality engineer agents) with a reflection agent for error handling - Orchestration system manages submission processing, requiring Proofs of Vulnerability (POVs) and validated patches. ### Vulnerability Discovery - Utilized standard OSS fuzzers (libfuzzer, jazzer) and an LLM input generation component to create bootstrapped and diverse inputs for fuzzing - LLM component also performed independent vulnerability discovery by exploring codebases and generating triggering inputs. ### Patching Process - Multi-agent system with programmatic data flow for control and guardrails - Agents retrieve context, determine root cause, plan and implement patches, and validate them against POVs and existing functionality - Relied on LLMs to unblock the process when agents failed. ### Competition Performance - Semi-finals saw Buttercup win 10 out of 42 achievements, indicating strong early performance - Finals required a rebuild (Buttercup 2.0) due to rule changes and increased scale (e.g., $50k LLM credits, $80k compute) - Secured second place in the finals, winning $3 million, with over 90% accuracy. ### Key to Success - High accuracy (over 90%) was critical for second place, outperforming a third-place team with 45% accuracy - Successful patching was worth significantly more points than simply finding vulnerabilities - Buttercup covered ground others did not, finding at least one POV no one else did. ### Open Source Release - Trail of Bits open-sourced both semi-final and final versions of Buttercup - A scaled-down standalone variant is available for laptop use, capable of finding and patching vulnerabilities within an hour - The team intends to maintain the open-source project using a portion of their winnings.