Foundation models for autonomous driving

Quick Overview

Waymo's autonomous driving technology achieves superhuman sensing ability by fusing multimodal data (vision, Lidar, Radar) and employing Large Multimodal Models (LMMs) based on Gemini architecture to accurately predict complex scenarios and align with real-world human safety preferences, resulting in significant reductions in crashes involving vulnerable road users compared to human drivers.

Key Points: Waymo's system achieves superhuman sensing ability by integrating vision, Lidar, and Radar data using a Large Multimodal Model based on Gemini. The model treats driving as a conversation, using trajectories as sentences and motion points as vocabulary, leveraging local continuity and global context. Post-training alignment using expert preferences (Inverse Reinforcement Learning) corrects pre-trained model behaviors, such as yielding to pedestrians and adopting human-like stopping behavior. Compared to human drivers over 56.3 million miles, Waymo Driver had 92% fewer crashes with injuries to pedestrians, 82% fewer with injuries to cyclists, and 82% fewer with injuries to motorcyclists. The system reduces the frequency of property damage claims by 88% and bodily injury claims by 92% based on third-party evaluation. Waymo is actively expanding operations, planning to launch in Miami next year and scanning many other cities for future deployment, including Japan. The approach focuses on aligning the AI with real-world safety objectives (safety, comfort, compliance) rather than simply imitating human driving.

Context: The presentation focuses on Waymo's advancements in autonomous driving AI, specifically detailing how they achieve safety and reliability, particularly when dealing with complex, long-tail scenarios. The speaker, an engineer from Waymo, introduces the use of multimodal AI models, specifically those based on the Gemini architecture, to process sensor data (Lidar, vision, radar) and textual context (like high-level commands or road sign interpretations) to generate accurate trajectories and ensure safety alignment with human preferences.