# On the Limits of Innate Planning in Large Language Models Source: https://www.youtube.com/watch?v=tVxuFBJaOXA Recap page: https://rapidrecap.app/video/tVxuFBJaOXA Generated: 2025-12-05T00:34:14.853+00:00 --- ## Quick Overview Large Language Models (LLMs) fundamentally struggle with long-term planning because they rely on predicting the next token rather than maintaining a consistent internal state, as demonstrated by their failure to solve the 8-puzzle consistently without explicit guidance, often resorting to looping or taking costly, counterproductive moves. **Key Points:** - LLMs struggle with planning tasks like the 8-puzzle, failing to maintain a perfect picture of the board state, leading to invalid moves. - The 8-puzzle, which requires 31 moves for the optimal solution, was frequently attempted by models like GPT-5 thinking, resulting in non-optimal moves or getting stuck in loops. - When given explicit step-by-step instructions (Chain-of-Thought), GPT-5 thinking achieved 68% success, while the smaller GPT-5 Mini achieved 30% success. - The core issue is that LLMs lack a reliable internal state tracker, causing them to forget the board state and make moves that violate simple rules. - When tested on the 8-puzzle with only the prompt (zero-shot), all tested models, including the larger ones, solved zero puzzles. - The research suggests that the fundamental limitation is the architecture's reliance on sequential token prediction rather than robust, recursive planning mechanisms. - The cost of solving one 8-puzzle instance via API calls was estimated to be around $75,284 tokens, highlighting the inefficiency of brute-force methods. ![Screenshot at 00:14: The video explicitly states that state-of-the-art LLMs achieve incredible results in areas that used to be difficult for humans, setting up the contrast with planning failures.](https://ss.rapidrecap.app/screens/tVxuFBJaOXA/00-00-14.png) **Context:** This video discusses the inherent limitations of current Large Language Models (LLMs) in performing complex, sequential planning tasks, using the classic 8-puzzle as a benchmark. The 8-puzzle requires maintaining a consistent, internal representation of the board state across multiple steps to find the optimal solution. The speakers contrast the performance of different models (GPT-5 thinking, GPT-5 Mini) under various prompting strategies (zero-shot, Chain-of-Thought) to illustrate this fundamental architectural constraint. ## Detailed Analysis The core finding of the research discussed is that LLMs fundamentally lack the ability to maintain a consistent internal state required for complex, long-term planning, which manifests as a failure in solving simple problems like the 8-puzzle. When given the 8-puzzle in a zero-shot setting, all models, including advanced ones like GPT-5 thinking, failed completely, resulting in zero solutions solved. When Chain-of-Thought (CoT) prompting was applied, GPT-5 thinking achieved 68% success, and GPT-5 Mini achieved 30% success. Crucially, the most powerful model, GPT-5 thinking, still failed entirely when relying solely on its raw, unguided reasoning (no external planning tools), suggesting a deep architectural flaw. When researchers tried to remedy this by providing explicit step-by-step instructions (the external move validator approach), the performance jumped to 100% success, proving the issue is planning, not raw intelligence. The failure mode involved loops, moving tiles to invalid locations, or simply forgetting the board state. The cost of failure is high, with one puzzle taking an estimated 75,284 tokens to solve via brute force, and the discussion concludes that future AI development must focus on new architectures or mechanisms beyond just larger models to achieve reliable, long-term autonomy. ### The 8-Puzzle Challenge - The 8-puzzle requires 31 moves for an optimal solution - Simple prompting (zero-shot) resulted in 0% success across models - The models exhibit a fundamental weakness in tracking internal state. ### Prompting Strategies and Results - GPT-5 thinking achieved 68% success with Chain-of-Thought (CoT) prompting - GPT-5 Mini achieved 30% success with CoT - GPT-5 thinking failed completely (0%) when relying only on raw, unguided reasoning. ### Failure Modes Identified - Models got stuck in loops, made illegal moves (moving tiles to empty space or off the board), and exhibited false confidence in incorrect solutions. ### The Role of External Tools - When models were guided by an external move validator (explicit feedback), success reached 100%, isolating the planning deficit. ### Cost Analysis - Solving one 8-puzzle instance using brute-force API calls cost approximately 75,284 tokens, indicating high expense for unguided planning. ### Conclusion on Future AI - The fundamental limitation is the architecture's reliance on next-token prediction rather than true, recursive planning, suggesting new architectures are needed for reliable AI autonomy. ![Screenshot at 00:00: The video opens with an image of two podcasters and the text "BECOME A MEMBER TODAY!" over a radar-like background, indicating the content is likely an AI-related discussion or podcast.](https://ss.rapidrecap.app/screens/tVxuFBJaOXA/00-00-00.png) ![Screenshot at 00:25: A visual representation of the 8-puzzle grid is shown, which is the central problem used to test LLM planning capabilities.](https://ss.rapidrecap.app/screens/tVxuFBJaOXA/00-00-25.png) ![Screenshot at 00:50: The speaker lists the models tested, including Gemini 2.5 Pro, GPT-4 Mini, and Llama 3.1 8B Instruct, providing context for the performance comparison.](https://ss.rapidrecap.app/screens/tVxuFBJaOXA/00-00-50.png) ![Screenshot at 01:16: A visual representation of the 8-puzzle failure is implied as the speaker discusses why LLMs fail at planning, referencing the need to pinpoint failures.](https://ss.rapidrecap.app/screens/tVxuFBJaOXA/00-01-16.png) ![Screenshot at 02:27: The speaker confirms that the models were given the easy puzzle without any external help, setting up the baseline failure rate.](https://ss.rapidrecap.app/screens/tVxuFBJaOXA/00-02-27.png)