# Querying Labeled Time Series Data with Scenario Programs Source: https://www.youtube.com/watch?v=Scv49i9Gyy0 Recap page: https://rapidrecap.app/video/Scv49i9Gyy0 Generated: 2025-11-17T22:35:09.311+00:00 --- ## Quick Overview The formal method of validating safety checks, which involves rigorously checking the syntactic structure against the program's logic, proves significantly more effective and less computationally expensive than relying solely on the pure, often ambiguous, semantic data from a simulated environment, especially when dealing with complex, state-machine governed systems like autonomous driving scenarios. **Key Points:** - The formal method guarantees safety validation by checking syntactic structure against program logic, unlike semantic data which can be ambiguous. - The formal method is significantly faster and less resource-intensive than physically testing every possible scenario in the real world. - The paper demonstrated the formal method's superiority by testing it against GPT-4o and Claude 3.5 Sonnet on four driving scenarios. - The formal method achieved an 80% accuracy in identifying correct actions, while GPT-4o scored 45% and Claude 3.5 Sonnet scored 60% for the same task. - The formal method successfully validated that the system's behavior matched the ground truth for actions like lane changes and intersections, avoiding the ambiguity of raw sensor data. - The core checking process in the formal method is fast and linear, contrasting sharply with the exponential complexity of exhaustively checking all possible state transitions. ![Screenshot at 08:18: Demonstration of the formal method's check: the system successfully verifies a potential scenario pairing \(two cars interacting\) by checking the formal logic against the observed behavior trace, contrasting with the ambiguity of raw sensor data.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-08-18.png) **Context:** This video discusses advancements in verifying the safety of autonomous systems, particularly self-driving cars, by comparing a formal verification method against large language models (LLMs) like GPT-4o and Claude 3.5 Sonnet when analyzing labeled time series data from simulations. The core issue addressed is the difficulty of ensuring that complex, real-world behaviors are correctly captured and predicted by algorithmic models trained on simulated data, especially when those simulations involve many interacting agents. ## Detailed Analysis The discussion centers on the superior reliability of a formal verification method for safety-critical systems compared to using large language models (LLMs) on simulated data. The formal approach checks the syntactic structure of the program logic directly, providing a guarantee that the system adheres to safety constraints, whereas relying on semantic data from simulations (like sensor readings or object positions) introduces ambiguity and potential for error. The paper demonstrated this by testing the formal method against GPT-4o and Claude 3.5 Sonnet using four driving scenarios involving two cars. The formal method achieved 80% accuracy in correctly identifying actions, significantly outperforming GPT-4o (45%) and Claude 3.5 Sonnet (60%). The formal method's advantage lies in its speed and precision; it runs linearly, checking state transitions iteratively, whereas exhaustive simulation testing is computationally exponential. Furthermore, the formal method provides a guaranteed correct match between the formal specification and the real-world scenario, something the LLMs struggled with due to the complexity of urban driving semantics. ### The Problem - Simulation vs. Reality: The foundation of safety for self-driving cars relies on simulation, but simulated sensors aren't realistic enough, leading to the need for rigorous validation of simulation fidelity against reality. ### Formal Method Superiority - The formal method rigorously checks syntactic structure against program logic, offering a guaranteed correctness check, unlike LLMs which rely on potentially ambiguous semantic data. ### Performance Comparison - Formal method achieved 80% accuracy across four scenarios, compared to GPT-4o at 45% and Claude 3.5 Sonnet at 60%, highlighting the gap between formal guarantees and LLM probabilistic guessing. ### Core Mechanism - State Machines and Guards: State machines govern system behavior, and the formal check verifies that if a safety condition (guard) is met (e.g., distance between cars < 3.5m), the prescribed action (lane change) occurs, proving the algorithm's logic is sound. ### Efficiency and Scalability - The formal search is fast and linear, avoiding the computational explosion (exponential complexity) that would result from trying to exhaustively test every possible state combination manually or through simulation. ![Screenshot at 04:04: Explaining the core issue: the immense complexity of modeling real-world events like a car changing lanes due to a slower car ahead.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-04-04.png) ![Screenshot at 05:15: The speaker emphasizes that the challenge is not the length of the simulation trace, but the number of interacting objects \(cars\) that must be simultaneously tracked.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-05-15.png) ![Screenshot at 06:36: Illustrating the formal method's basis: checking if the base states of the state machines \(Follow Lane, Turn Left\) match the observed behavior.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-06-36.png) ![Screenshot at 07:25: Detailing the formal check: the system checks the syntax against the logic, guaranteeing that the outcome matches the expected behavior for the specific time slice.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-07-25.png) ![Screenshot at 08:18: Visualizing the validation step where the formal system correctly identifies a potential scenario pairing \(two cars interacting\) that the LLMs struggled to match perfectly.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-08-18.png) ![Screenshot at 09:38: Comparing the accuracy results: Formal method \(80%\) significantly outperforms GPT-4o \(45%\) and Claude 3.5 Sonnet \(60%\) on the same test set.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-09-38.png) ![Screenshot at 10:18: Showing the time difference: LLMs took over 40 seconds for complex scenarios, whereas the formal method was much faster on a linear scale.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-10-18.png) ![Screenshot at 11:52: The speaker concludes that the formal approach is a fast, linear, and reliable method that avoids the computational wall faced by exhaustive testing.](https://ss.rapidrecap.app/screens/Scv49i9Gyy0/00-11-52.png)