# Testing 1, 2, 3 Testing: Automatically Finding and Fixing Software Vulnerabilities at Scale & Speed Source: https://www.youtube.com/watch?v=Ty89elXvDdE Recap page: https://rapidrecap.app/video/Ty89elXvDdE Generated: 2025-09-26T16:06:24.245+00:00 --- ## Quick Overview The presentation details the LACROSSE system, a cyber reasoning security system developed by SIFT, which leverages LLMs to automate vulnerability identification and patching, showcasing its pipeline testing and prompts. **Key Points:** - The LACROSSE system is a cyber reasoning security system developed by SIFT that integrates LLMs to automate vulnerability identification and patching. - The system was presented by Dr. David Musliner and Dr. Matt McLure from SIFT at DEF CON 33. - LACROSSE builds upon SIFT's prior work in multi-agent cyber reasoning systems, dating back to DARPA projects in 2000. - The system uses a pipeline approach involving identifying vulnerabilities, filtering and scoring them, and then attempting to patch them. - LLMs are used in various stages, including generating prompts, rewriting code snippets, and assessing vulnerability likelihood. - The pipeline includes stages like 'Identify Vulnerabilities', 'Filter & Score', 'Refine Vulnerability Hypotheses', and 'Patch'. - The system is designed to handle rate limits and failures by retrying operations with different LLMs or using back-off schedules. ![Screenshot at 00:17: The title slide of the presentation, "Testing 1, 2, 3 Testing: Automatically Finding and Fixing Software Vulnerabilities at Scale and Speed," lists the speakers Dr. David Musliner and Dr. Matt McLure from SIFT, and the event date August 8, 2025.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-00-17.png) **Context:** This presentation, "Testing 1, 2, 3 Testing: Automatically Finding and Fixing Software Vulnerabilities at Scale & Speed," delivered by Dr. David Musliner and Dr. Matt McLure from SIFT at DEF CON 33, introduces LACROSSE, a novel cyber reasoning security system. LACROSSE integrates Large Language Models (LLMs) into the process of identifying and fixing software vulnerabilities, building on SIFT's extensive experience in AI and cybersecurity research. ## Detailed Analysis The presentation introduces LACROSSE, a cyber reasoning security system developed by SIFT, which leverages Large Language Models (LLMs) to automate the process of finding and fixing software vulnerabilities. The system builds on SIFT's history of work in this area, starting with DARPA projects in 2000. The core of LACROSSE involves a pipeline that includes identifying vulnerabilities from code, filtering and scoring these raw hypotheses, and refining them into actionable insights. LLMs play a crucial role in various stages, such as generating prompts for vulnerability analysis, rewriting code snippets to create patches, and providing descriptive explanations for identified vulnerabilities. The system handles failures and rate limits by implementing retry mechanisms, including swapping to different LLMs or using back-off schedules. A key feature is the use of parallel consensus, where multiple LLMs operate in parallel, and their outputs are aggregated to pick the most representative result, enhancing robustness and reducing hallucinations. The presentation also touches upon the challenges of prompt engineering and the importance of context for LLMs in accurately assessing vulnerabilities. The pipeline testing approach aims to achieve faster and more reliable vulnerability detection and patching, showcasing a significant advancement in automated cybersecurity. ### Introduction - SIFT and LACROSSE - SIFT is a 35-person small business focused on research in CS+Humans, with 25 years in business and distributed US offices - R&D expertise includes Computer Science, AI, Psychology, with over 400 years of combined industrial R&D experience - Research products transitioned to IC, State Dept, commercial applications ### SIFTies at DEFCON - Presentation features five SIFT team members: Luis Baez, Matt McLure, David Musliner, Nathan Ringo, and Dan Thomsen ### SIFT Cyber Reasoning System (CRS) History - Started in 2000 with DARPA Cyber Panel/Cortex, evolved through DARPA CRASH/Fuzzbuster (2009), DARPA CGC/Fuzzbomb (2014), and ONR NEOFuzz (2016) - Current LACROSSE system integrates LLM+AI for cyber reasoning security ### LACROSSE - LLM+AI Cyber Reasoning Security System - Combines existing multi-agent CRS with Code-trained LLMs - Features include LLM vulnerability detection, LLM patch generation, and LLM reasoning ### LLM-based Tasks - Includes tasks like hypothesize-vulns, hypothesize-seeds, assess-sarif, fuzz, explain-pov, patch-sans-pov, patch-with-pov - Uses LLM for generating patches and identifying vulnerabilities ### Characterization Pipeline - Input (Select Files/Functions, Code Snippets, SARIF) -> Identify Vulnerabilities -> Raw Vulnerability Hypotheses -> Filter & Score -> Refined Vulnerability Hypotheses - Also includes Crash log filtering and Delta analysis ### Prompts and Mixins - Demonstrates the structure of prompts and mixins used in the system, highlighting the use of DSPY for signature, input, and output fields ### Pipeline Testing - Evaluated pipeline approaches in a tighter testing loop (~3 min vs. ~30 min) - 4 levels of success per problem: Apply, Build, Fix PoV, Pass Functionality tests - Nginx dataset stack graph provided per-problem results, allowing identification of complementary LLM approaches - Tested LLMs on different tasks and models for accuracy and performance ![Screenshot at 00:00: Title slide for 'AIxCC Stage at DEF CON 33' with the AIxCC logo.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-00-00.png) ![Screenshot at 00:17: The presentation title slide, 'Testing 1, 2, 3 Testing: Automatically Finding and Fixing Software Vulnerabilities at Scale and Speed,' lists the speakers and date.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-00-17.png) ![Screenshot at 01:06: Slide detailing 'Who is SIFT?' with information about the company's focus, history, locations, and R&D expertise.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-01-06.png) ![Screenshot at 01:33: Slide titled 'SIFTies at DEFCON' shows profile pictures and names of five team members: Luis Baez, Matt McLure, David Musliner, Nathan Ringo, Dan Thomsen.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-01-33.png) ![Screenshot at 01:48: Slide titled 'SIFT Cyber Reasoning System \(CRS\) History' outlines the evolution of their systems from 2000 to the present LACROSSE system.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-01-48.png) ![Screenshot at 03:17: Diagram illustrating the 'LACROSSE: LLM+AI Cyber Reasoning Security System' architecture, showing interaction between LLMs and analysis tools.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-03-17.png) ![Screenshot at 05:21: Diagram showing the 'Cross-Cueing Between Analysis Tools' workflow, highlighting how different tools and LLMs interact.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-05-21.png) ![Screenshot at 08:04: Slide summarizing 'Recent LLM Advancements,' including improved code understanding, reduced prompt engineering needs, and reliability improvements.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-08-04.png) ![Screenshot at 09:38: Diagram illustrating 'LLM-based Tasks' within the LACROSSE system, showing various task dependencies.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-09-38.png) ![Screenshot at 12:12: Diagram detailing the 'Characterization' pipeline, from selecting files to identifying vulnerabilities and refining hypotheses.](https://ss.rapidrecap.app/screens/Ty89elXvDdE/00-12-12.png)