# The 2025 Planning Performance of Frontier Large Language Models

Source: https://www.youtube.com/watch?v=YpD6XJ8AA0A
Recap page: https://rapidrecap.app/video/YpD6XJ8AA0A
Generated: 2025-11-15T15:06:17.028+00:00

---
## Quick Overview

The 2025 planning performance evaluation reveals that specialized LLMs like DeepSeek R1 and Google's Gemini 2.5 Pro significantly outperform larger, more general models like GPT-5 on planning tasks, despite GPT-5 showing superior raw coverage on standard benchmarks, indicating that focused training and specialized architectures are crucial for complex, real-world planning.

**Key Points:**
- DeepSeek R1 and Gemini 2.5 Pro outperformed GPT-5 on planning tasks from the IPC 2023 learning track, solving 205 and 157 tasks respectively, compared to GPT-5's 152 solved tasks.
- GPT-5 showed a steep performance drop (from 157 to 93 solved tasks) when tested on obfuscated planning problems, suggesting a reliance on superficial cues rather than true symbolic reasoning.
- The classical planner, Lama, ran on a single CPU, required 1000 Gib of memory for inference, and solved 157 tasks, demonstrating the computational cost of non-LLM approaches.
- The cost to run GPT-5 for the evaluation was only $100, whereas the computational cost for the specialized planners was essentially free, highlighting an efficiency trade-off.
- The core difference is that specialized LLMs like DeepSeek R1 and Gemini 2.5 Pro demonstrated better robustness by relying less on semantic cues and more on pure symbolic logic.
- The ability of these specialized models to solve complex real-world problems like logistics and self-driving car planning is a major achievement for the frontier LLMs, despite the cost/resource trade-off.

![Screenshot at 03:37: The visual highlights the core finding: the performance gap between the specialized LLMs and GPT-5 on the standard PDL domain, marking a significant milestone for specialized planning within LLMs.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-03-37.png)

**Context:** This video from the AI Papers Podcast reviews the 2025 planning evaluation results, comparing the performance of large language models (LLMs) like GPT-5 against specialized planning systems such as DeepSeek R1 and Gemini 2.5 Pro on the International Planning Competition (IPC) learning track. The evaluation focused on the models' ability to generate valid sequences of actions to move from an initial state to a goal state in complex domains.

## Detailed Analysis

The discussion centers on the 2025 planning evaluation results, where specialized LLMs demonstrated superior capability in generating valid plans compared to GPT-5. DeepSeek R1 and Gemini 2.5 Pro achieved significantly higher success rates on IPC tasks (205 and 157 solved tasks, respectively) than GPT-5 (152 solved tasks). Furthermore, when tested on obfuscated symbolic names, GPT-5's performance dropped severely, indicating it relied heavily on surface-level semantic cues rather than true logical structure, whereas the specialized models maintained robustness. The performance gap is stark: GPT-5's performance dropped by orders of magnitude when moving from standard to obfuscated tasks. The classical planner Lama required massive resources (1000 GiB of memory on a single CPU) to achieve performance comparable to the LLMs. The ultimate conclusion is that while GPT-5 is a powerful generalist, focused architectures and training (like those in R1 and Gemini 2.5 Pro) provide a significant edge in complex, real-world planning tasks that require pure symbolic logic, suggesting that specialized reasoning remains critical.

### 2025 Planning Evaluation Summary

- DeepSeek R1 solved 205 tasks, Gemini 2.5 Pro solved 157 tasks, and GPT-5 solved 152 tasks on the IPC 2023 learning track
- Performance gap is significant despite GPT-5 being larger and more general

### Impact of Obfuscation

- GPT-5's performance dropped drastically when symbolic names were replaced with random strings (e.g., 'truck A to B' became 'XTRN7QZ4H')
- This reveals a reliance on semantics over pure logic

### Resource Comparison

- Classical planner Lama required 1000 GiB of memory on a single CPU to solve 157 tasks
- GPT-5 evaluation cost was only $100, but specialized LLMs showed better results for less resource-intensive setups

### Key Takeaway

- LLMs must master generalized symbolic logic rather than just relying on semantic shortcuts
- The performance gap in reliability for long-range planning between specialized and general models remains evident

![Screenshot at 00:01: Visual introduction screen with podcast branding and call to action.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-00-01.png)
![Screenshot at 00:10: The speakers discuss the general capabilities of large language models.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-00-10.png)
![Screenshot at 00:33: The host poses the core question about what AI planning is.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-00-33.png)
![Screenshot at 00:49: The host summarizes the goal: providing a shortcut to understanding planning performance.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-00-49.png)
![Screenshot at 01:38: Visual of the IPC 2023 learning track being mentioned as the standard for testing.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-01-38.png)
![Screenshot at 02:49: The speaker defines PDDL \(Planning Domain Definition Language\) as the formal system used for validation.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-02-49.png)
![Screenshot at 03:37: A visual representation of the performance drop on the PDL domain for GPT-5 compared to specialized models.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-03-37.png)
![Screenshot at 04:48: The speaker mentions the first large model failure on the Floor Tile domain.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-04-48.png)
![Screenshot at 07:23: Visual representation of the massive computational resources required by the classical planner Lama.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-07-23.png)
![Screenshot at 09:00: The speaker discusses the hardware requirements \(24 GB GPU\) needed to run models locally.](https://ss.rapidrecap.app/screens/YpD6XJ8AA0A/00-09-00.png)