AI GAMESTORE: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games

Quick Overview

The paper introduces AI GameStore, a framework for evaluating general machine intelligence by pitting AI models against human players in a curated set of games, revealing that while models like GPT-4.5 and Claude 4.5 are proficient at coding and logic, they severely lack real-world understanding, flow, and planning capabilities compared to humans, especially in dynamic, non-explicitly programmed environments.

Key Points: AI GameStore proposes a radical shift in measuring machine intelligence away from static tests toward dynamic evaluation using human games. GPT-4.5 and Claude 4.5 showed strong performance (90s percentile) on coding and math benchmarks but failed completely (0% progress) on tasks requiring real-world understanding, such as learning the rules of a simple 2D platformer. The models struggled to infer hidden rules through interaction, especially in environments requiring planning or adapting to game mechanics that weren't explicitly programmed. The research highlights a severe cognitive gap: models excel at retrieving crystallized knowledge but fail at fluid intelligence required for real-time, dynamic environments like games. The study suggests the AI GameStore concept provides a more dynamic, yet computationally expensive, benchmark than traditional static tests. The researchers created a curated set of 100 polished games from platforms like the Apple App Store and Steam for testing.

Context: The video discusses a research paper titled "AI GameStore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games," which proposes a new method for testing Artificial General Intelligence (AGI). This method moves beyond traditional, static benchmarks by observing how AI models perform when playing various human-created video games, including puzzle, strategy, and simulation genres, to gauge their real-world adaptability and understanding.

Detailed Analysis

The paper argues for a significant shift in how machine intelligence is measured, moving away from static tests to dynamic evaluation using human games via a framework called AI GameStore. This approach tests an AI's ability to generalize, learn, and adapt in real-time, dynamic environments. The researchers tested models including GPT-4.5 and Claude 4.5 on 100 polished games scraped from sources like the Apple App Store and Steam. While these large language models (LLMs) demonstrated high scores (90th percentile) on crystallized knowledge tests like programming syntax and math benchmarks, they performed extremely poorly when tasked with learning and playing these games, often failing completely (0% progress) on simple 2D platformers. This failure highlights a major cognitive gap: the models understand explicit code but cannot infer implicit rules or exhibit the necessary planning and flow required for real-world interaction. For instance, the models failed to understand the simple physics of a game where an agent falls into a pit, suggesting a lack of common sense. The researchers noted that the models struggled significantly more than human players, who rely on intuition and rhythm. The study suggests that future AGI evaluation must focus on these dynamic, interactive benchmarks rather than relying solely on static tests or memorized patterns.