# TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents Source: https://www.youtube.com/watch?v=2omJGI01r-g Recap page: https://rapidrecap.app/video/2omJGI01r-g Generated: 2026-01-18T20:04:25.093+00:00 --- ## Quick Overview The TowerMind environment successfully quantifies the performance of Large Language Models (LLMs) as agents in a dynamic Tower Defense setting, revealing a significant performance gap where larger models like GPT-4.1 and Claude 3.5 Sonnet outperform smaller open-source models and struggle with complex planning, despite showing good tactical adherence to rules. **Key Points:** - TowerMind is introduced as a new Tower Defense game learning environment and benchmark for evaluating LLMs as agents (0:21). - The environment tests LLMs' ability to handle complex, real-time decision-making involving resource management, base defense, and unit management (2:37, 3:03). - GPT-4.1 and Claude 3.5 Sonnet significantly outperformed all smaller open-source models, scoring 62% versus the human expert average of 90.96% (7:34, 9:09). - Smaller open-source models performed substantially worse than the human baseline, with some models scoring worse than random guessing (9:57, 10:00). - A key failure point is the models' inability to connect tactical actions (like building a tower) with long-term strategic implications, leading to poor resource allocation (11:17, 12:54). - The experiment involved testing across five difficulty levels, with performance degrading significantly as complexity increased (6:57, 8:03). - The multimodal models (like GPT-4V) performed better than text-only models when dealing with visual input (2:24, 8:24). ![Screenshot at 0:21: Introduction of TowerMind as a Tower Defense game learning environment and benchmark for LLM agents.](https://ss.rapidrecap.app/screens/2omJGI01r-g/00-00-21.jpg) **Context:** The video introduces TowerMind, a novel testing environment designed specifically to benchmark Large Language Models (LLMs) acting as autonomous agents within a Real-Time Strategy (RTS) Tower Defense game context. This environment forces agents to manage resources, plan defensively, and react to dynamic threats, providing a critical test case for strategic reasoning beyond simple text completion. The evaluation focuses on how well LLMs can synthesize complex game state information and execute long-term plans, often failing when required to balance immediate tactical needs against strategic resource allocation. ## Detailed Analysis The TowerMind environment serves as a benchmark for testing LLM agents in a complex, dynamic Tower Defense game setting. The fundamental problem addressed is ensuring LLMs can perform reliable, real-time decision-making, specifically in areas like healthcare tasks and complex strategy games, which traditionally require robust long-term planning. The agents must master two things: planning and robust real-time decision-making. The game map is a simple square area, where enemies follow fixed paths towards the base, starting with 20 health. The primary tactical skills tested involve base defense, unit management, and resource allocation, forcing agents to choose between fighting or gathering gold (4:52). A major limitation observed is the failure of agents to link tactical actions to long-term strategic outcomes, exemplified by agents building towers in strategically useless locations (8:44). The evaluation metrics focus on the ratio of valid actions to total actions, where a score of zero indicates perfect defense. GPT-4.1 and Claude 3.5 Sonnet achieved the highest scores, but still showed significant gaps compared to human experts (7:51). Furthermore, the visual input modality (like GPT-4V) proved superior to text-only models in handling the complex visual information (8:24). The research concludes that the inherent complexity and the need for multi-finality thinking (achieving multiple goals simultaneously) remains a major barrier for current LLMs in complex strategic environments. ### TowerMind Environment Introduction - Takes large language models to test complex tasks from healthcare to Tower Defense (0:03) - The core challenge is creating reliable LLMs for real-time decision making (0:28) - The environment is designed to test planning and robust real-time tactical changes (0:36, 0:57). ### Resource Management and Strategy - Agents must balance choosing between defense (fighting) or economy (collecting gold) (4:50, 5:04) - The map is a simple square area, with enemies following fixed paths towards the base (3:56, 4:00). ### Performance Metrics and Results - The core metric is the ratio of valid actions to total actions; zero is a perfect score (6:17, 6:26) - GPT-4.1 and Claude 3.5 Sonnet performed best (62% average score) compared to human experts (90.96%) (7:35, 9:09) - Smaller open-source models significantly underperformed, sometimes worse than random guessing (9:57). ### Key Limitations Identified - Failure to link tactical choices to long-term strategy, leading to inefficient resource use (11:17, 12:54) - LLMs often misuse critical resources (12:12) and fail to update strategy based on dynamic map states (5:14). ### Multimodality Advantage - Vision-language models performed better than text-only models, suggesting visual input is crucial for understanding the game state (8:24). ![Screenshot at 0:21: The introduction slide defining the paper's focus on TowerMind as an LLM agent benchmark \(0:21\).](https://ss.rapidrecap.app/screens/2omJGI01r-g/00-00-21.jpg) ![Screenshot at 3:56: Illustration of the simple, square map area used in the TowerMind environment \(3:56\).](https://ss.rapidrecap.app/screens/2omJGI01r-g/00-03-56.jpg) ![Screenshot at 7:35: Data visualization showing the comparison between commercial LLMs \(like GPT-4.1\) and human expert performance scores \(7:35\).](https://ss.rapidrecap.app/screens/2omJGI01r-g/00-07-35.jpg) ![Screenshot at 8:24: Comparison highlighting that multimodal models \(vision\) perform better than text-only models in this complex environment \(8:24\).](https://ss.rapidrecap.app/screens/2omJGI01r-g/00-08-24.jpg) ![Screenshot at 10:09: Speaker introduces the 'qualitative limitations' section, focusing on strategic failures \(10:09\).](https://ss.rapidrecap.app/screens/2omJGI01r-g/00-10-09.jpg)