The 2025 Planning Performance of Frontier Large Language Models

Quick Overview

The 2025 planning performance evaluation reveals that specialized LLMs like DeepSeek R1 and Google's Gemini 2.5 Pro significantly outperform larger, more general models like GPT-5 on planning tasks, despite GPT-5 showing superior raw coverage on standard benchmarks, indicating that focused training and specialized architectures are crucial for complex, real-world planning.

Key Points: DeepSeek R1 and Gemini 2.5 Pro outperformed GPT-5 on planning tasks from the IPC 2023 learning track, solving 205 and 157 tasks respectively, compared to GPT-5's 152 solved tasks. GPT-5 showed a steep performance drop (from 157 to 93 solved tasks) when tested on obfuscated planning problems, suggesting a reliance on superficial cues rather than true symbolic reasoning. The classical planner, Lama, ran on a single CPU, required 1000 Gib of memory for inference, and solved 157 tasks, demonstrating the computational cost of non-LLM approaches. The cost to run GPT-5 for the evaluation was only $100, whereas the computational cost for the specialized planners was essentially free, highlighting an efficiency trade-off. The core difference is that specialized LLMs like DeepSeek R1 and Gemini 2.5 Pro demonstrated better robustness by relying less on semantic cues and more on pure symbolic logic. The ability of these specialized models to solve complex real-world problems like logistics and self-driving car planning is a major achievement for the frontier LLMs, despite the cost/resource trade-off.

Context: This video from the AI Papers Podcast reviews the 2025 planning evaluation results, comparing the performance of large language models (LLMs) like GPT-5 against specialized planning systems such as DeepSeek R1 and Gemini 2.5 Pro on the International Planning Competition (IPC) learning track. The evaluation focused on the models' ability to generate valid sequences of actions to move from an initial state to a goal state in complex domains.

Detailed Analysis

The discussion centers on the 2025 planning evaluation results, where specialized LLMs demonstrated superior capability in generating valid plans compared to GPT-5. DeepSeek R1 and Gemini 2.5 Pro achieved significantly higher success rates on IPC tasks (205 and 157 solved tasks, respectively) than GPT-5 (152 solved tasks). Furthermore, when tested on obfuscated symbolic names, GPT-5's performance dropped severely, indicating it relied heavily on surface-level semantic cues rather than true logical structure, whereas the specialized models maintained robustness. The performance gap is stark: GPT-5's performance dropped by orders of magnitude when moving from standard to obfuscated tasks. The classical planner Lama required massive resources (1000 GiB of memory on a single CPU) to achieve performance comparable to the LLMs. The ultimate conclusion is that while GPT-5 is a powerful generalist, focused architectures and training (like those in R1 and Gemini 2.5 Pro) provide a significant edge in complex, real-world planning tasks that require pure symbolic logic, suggesting that specialized reasoning remains critical.