how this tiny model beat ChatGPT on the “AGI” benchmark [HRM & TRM]

Quick Overview

The Tiny Recursion Model (TRM) significantly outperforms the larger Hierarchical Reasoning Model (HRM) and surpasses current state-of-the-art Large Language Models (LLMs) on the ARC-AGI benchmark by employing a simpler, recursive, attention-free architecture with only 7 million parameters, achieving 40% accuracy on ARC-AGI-1 for only $1.76 per task.

Key Points: The Tiny Recursion Model (TRM), with only 7 million parameters, achieved 40% accuracy on ARC-AGI-1 and 6.2% on ARC-AGI-2, significantly beating larger LLMs like GPT-4 and Claude. TRM outperforms the Hierarchical Reasoning Model (HRM) on Sudoku-Extreme (87.4% vs 55.0%) and Maze-Hard (85.3% vs 74.5%), despite TRM being 4x smaller than HRM (7M vs 27M parameters). The HRM paper's initial success relied on heuristics like assuming the low-level network settled at equilibrium during training, which the TRM paper shows leads to inaccurate gradients. TRM achieves its performance without complex mathematical theorems, hierarchy, or biological arguments, relying on a single, tiny network structure that recursively updates a scratchpad. The authors of TRM justify its design by showing that increasing layers in HRM architecture only leads to overfitting on scarce training data, indicating that scaling up is not always the answer. TRM achieves its results using only MLPs in its blocks (instead of attention layers) and a single forward pass, which is simpler and more efficient than the two-module, multi-timescale approach of HRM.

Context: This video analyzes the research behind the Tiny Recursion Model (TRM), a novel, small-scale AI architecture, contrasting it with the previous state-of-the-art, the Hierarchical Reasoning Model (HRM), particularly regarding performance on the Abstract and Reasoning Corpus (ARC) benchmark, often considered a test for Artificial General Intelligence (AGI). The video critiques the dominant scaling laws in AI, suggesting that for complex reasoning tasks with limited data, architectural simplicity and efficiency, as demonstrated by TRM, can surpass massive parameter counts.