The Cost of Thinking Is Similar Between Large Reasoning Models and Humans

Quick Overview

The reasoning cost for advanced Large Reasoning Models (LRMs) like DeepSeek R1 is remarkably similar to that of humans when solving complex reasoning tasks, as demonstrated by a 2024 study comparing computational effort to human cognitive effort across various tasks like arithmetic, syllogisms, and intuitive reasoning.

Key Points: A 2024 study by Davarda, Delia, and colleagues compared the computational cost of large reasoning models to human cognitive effort. The study focused on tasks including arithmetic, syllogisms, and intuitive reasoning problems. The reasoning cost for the LRM DeepSeek R1, when measured by reasoning token count, strongly correlated (R=0.97) with the average human reaction time for the same tasks. The base model, DeepSeek V3, showed an R correlation of 0.44 against human reaction time, indicating weaker alignment than R1. The study suggests that complex reasoning, like formal logic and intuitive problem-solving, requires a greater cognitive investment than simple arithmetic for both humans and advanced LRMs. The effort metric, which is the length of the chain of thought, was found to be a powerful metric for quantifying cognitive load across different problem types.

Context: This video discusses recent research comparing the cognitive effort required for artificial intelligence models to solve reasoning problems against the effort required by humans. The research specifically investigates how the computational cost (measured in tokens/steps) of advanced Large Reasoning Models (LRMs) aligns with human cognitive load, using a study that tested models like DeepSeek R1 and DeepSeek V3 on various reasoning tasks.

Detailed Analysis

The discussion centers on a 2024 study by Davarda, Delia, and colleagues that quantified the 'cost of thinking' for Large Reasoning Models (LRMs) and compared it to human cognitive effort. The researchers measured the cost in two primary ways: the length of the reasoning chain (chain of thought length) and the computational cost of the tokens generated. When testing the DeepSeek R1 model, they found an exceptionally strong correlation (R=0.97) between the model's reasoning token count and the average time humans took to solve the exact same problems, which included arithmetic, syllogisms, and intuitive reasoning. This suggests that the computational resources consumed by the LRM mirror human cognitive load. For comparison, the base model, DeepSeek V3, showed a weaker correlation (R=0.44). The study also controlled for the length of the prompt, showing that the correlation remained strong even when comparing complex abstract problems (like syllogisms) to simple arithmetic, where humans typically invest less cognitive effort. The researchers argue that this chain of thought metric is a powerful, quantifiable signal that mirrors human cognitive struggle, implying that advanced models are learning to reason in a human-like, effort-allocating manner rather than just rote memorization.