# AI GAMESTORE: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games Source: https://www.youtube.com/watch?v=vweMiN0YBJs Recap page: https://rapidrecap.app/video/vweMiN0YBJs Generated: 2026-02-23T18:34:06.857+00:00 --- ## Quick Overview The paper introduces AI GameStore, a framework for evaluating general machine intelligence by pitting AI models against human players in a curated set of games, revealing that while models like GPT-4.5 and Claude 4.5 are proficient at coding and logic, they severely lack real-world understanding, flow, and planning capabilities compared to humans, especially in dynamic, non-explicitly programmed environments. **Key Points:** - AI GameStore proposes a radical shift in measuring machine intelligence away from static tests toward dynamic evaluation using human games. - GPT-4.5 and Claude 4.5 showed strong performance (90s percentile) on coding and math benchmarks but failed completely (0% progress) on tasks requiring real-world understanding, such as learning the rules of a simple 2D platformer. - The models struggled to infer hidden rules through interaction, especially in environments requiring planning or adapting to game mechanics that weren't explicitly programmed. - The research highlights a severe cognitive gap: models excel at retrieving crystallized knowledge but fail at fluid intelligence required for real-time, dynamic environments like games. - The study suggests the AI GameStore concept provides a more dynamic, yet computationally expensive, benchmark than traditional static tests. - The researchers created a curated set of 100 polished games from platforms like the Apple App Store and Steam for testing. ![Screenshot at 00:00: The opening screen displays the podcast/video branding with an audio wave graphic and a clear call to action: "BECOME A MEMBER TODAY!"](https://ss.rapidrecap.app/screens/vweMiN0YBJs/00-00-00.jpg) **Context:** The video discusses a research paper titled "AI GameStore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games," which proposes a new method for testing Artificial General Intelligence (AGI). This method moves beyond traditional, static benchmarks by observing how AI models perform when playing various human-created video games, including puzzle, strategy, and simulation genres, to gauge their real-world adaptability and understanding. ## Detailed Analysis The paper argues for a significant shift in how machine intelligence is measured, moving away from static tests to dynamic evaluation using human games via a framework called AI GameStore. This approach tests an AI's ability to generalize, learn, and adapt in real-time, dynamic environments. The researchers tested models including GPT-4.5 and Claude 4.5 on 100 polished games scraped from sources like the Apple App Store and Steam. While these large language models (LLMs) demonstrated high scores (90th percentile) on crystallized knowledge tests like programming syntax and math benchmarks, they performed extremely poorly when tasked with learning and playing these games, often failing completely (0% progress) on simple 2D platformers. This failure highlights a major cognitive gap: the models understand explicit code but cannot infer implicit rules or exhibit the necessary planning and flow required for real-world interaction. For instance, the models failed to understand the simple physics of a game where an agent falls into a pit, suggesting a lack of common sense. The researchers noted that the models struggled significantly more than human players, who rely on intuition and rhythm. The study suggests that future AGI evaluation must focus on these dynamic, interactive benchmarks rather than relying solely on static tests or memorized patterns. ### Introduction to AI GameStore - Proposes a radical shift from static tests to dynamic evaluation using human games - Moves measurement toward fluid intelligence and real-world adaptability - Involves testing on 100 polished games from existing app stores and Steam. ### Model Performance Comparison - GPT-4.5 and Claude 4.5 scored highly (90s percentile) on traditional benchmarks (coding, math) - Showed 0% progress on many game tasks, failing to learn simple 2D platformer rules. ### Cognitive Deficiencies Identified - LLMs struggle with inferring implicit rules, planning, and temporal dynamics (like Flappy Bird) - Lack the fundamental understanding of physics and psychology needed for complex real-world interaction. ### Evaluation Methodology - The system uses a screenshot and text description of the game state every 0.2 seconds to prompt the AI, which then outputs a list of actions for the next 0.2 seconds. - This iterative process, called an evaluation harness, forces the AI to engage dynamically. ### Conclusion and Implications - The massive performance gap between static tests and dynamic games proves that current LLMs lack true general intelligence - The industry must pivot roadmaps toward developing agents capable of proactive planning and understanding real-world dynamics, not just pattern recall. ![Screenshot at 00:00: Podcast introduction screen with the title card "BECOME A MEMBER TODAY!"](https://ss.rapidrecap.app/screens/vweMiN0YBJs/00-00-00.jpg) ![Screenshot at 01:14: Visual representation of the evaluation process showing a gap between the models' performance \(95th percentile\) on static tests and their failure on dynamic tasks.](https://ss.rapidrecap.app/screens/vweMiN0YBJs/00-01-14.jpg) ![Screenshot at 02:26: Explanation of the evaluation harness where the system acts as an interface between the model and the game engine, pausing every 0.2 seconds for decision-making.](https://ss.rapidrecap.app/screens/vweMiN0YBJs/00-02-26.jpg) ![Screenshot at 05:56: Comparison of the model's performance on a specific game where it scored 106 human players against 7 frontier models, showing a massive performance gap.](https://ss.rapidrecap.app/screens/vweMiN0YBJs/00-05-56.jpg) ![Screenshot at 09:20: Final summary slide emphasizing that AI requires about 20 minutes to complete a task that a human can do in 2 minutes, highlighting the inefficiency.](https://ss.rapidrecap.app/screens/vweMiN0YBJs/00-09-20.jpg)