TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents

Quick Overview

The TowerMind environment successfully quantifies the performance of Large Language Models (LLMs) as agents in a dynamic Tower Defense setting, revealing a significant performance gap where larger models like GPT-4.1 and Claude 3.5 Sonnet outperform smaller open-source models and struggle with complex planning, despite showing good tactical adherence to rules.

Key Points: TowerMind is introduced as a new Tower Defense game learning environment and benchmark for evaluating LLMs as agents (0:21). The environment tests LLMs' ability to handle complex, real-time decision-making involving resource management, base defense, and unit management (2:37, 3:03). GPT-4.1 and Claude 3.5 Sonnet significantly outperformed all smaller open-source models, scoring 62% versus the human expert average of 90.96% (7:34, 9:09). Smaller open-source models performed substantially worse than the human baseline, with some models scoring worse than random guessing (9:57, 10:00). A key failure point is the models' inability to connect tactical actions (like building a tower) with long-term strategic implications, leading to poor resource allocation (11:17, 12:54). The experiment involved testing across five difficulty levels, with performance degrading significantly as complexity increased (6:57, 8:03). The multimodal models (like GPT-4V) performed better than text-only models when dealing with visual input (2:24, 8:24).

Context: The video introduces TowerMind, a novel testing environment designed specifically to benchmark Large Language Models (LLMs) acting as autonomous agents within a Real-Time Strategy (RTS) Tower Defense game context. This environment forces agents to manage resources, plan defensively, and react to dynamic threats, providing a critical test case for strategic reasoning beyond simple text completion. The evaluation focuses on how well LLMs can synthesize complex game state information and execute long-term plans, often failing when required to balance immediate tactical needs against strategic resource allocation.