# Energy-Based Transformers are Scalable Learners and Thinkers (Paper Review)

Source: https://www.youtube.com/watch?v=RAEy3JZmIaA
Recap page: https://rapidrecap.app/video/RAEy3JZmIaA
Generated: 2025-07-19T16:05:38.582+00:00

---
## Quick Overview

Energy-Based Transformers (EBTs) are introduced as a novel architecture that enables System 2 Thinking capabilities in models through unsupervised learning, demonstrating superior scalability and generalization compared to traditional Transformer models. EBTs achieve up to 35% higher scaling rates across data, batch size, and depth, and outperform Diffusion Transformers in image denoising with fewer forward passes, suggesting a promising new paradigm for advancing foundation models.

**Key Points:**
- Energy-Based Transformers (EBTs) enable System 2 Thinking in AI models through unsupervised learning, addressing limitations of current models in complex reasoning.
- EBTs learn an energy function that assigns compatibility scores to inputs and predictions, minimizing this energy through iterative gradient descent during inference.
- The architecture allows for dynamic computation allocation, modeling uncertainty in continuous state spaces, and explicit verification of predictions.
- EBTs demonstrate superior scalability, achieving up to 35% faster scaling rates than Transformer++ across data, batch size, and depth during pretraining.
- EBTs outperform Diffusion Transformers in image denoising, requiring 99% fewer forward passes for comparable or better results.
- The models show strong generalization capabilities, particularly on out-of-distribution data, and exhibit self-assessment of prediction uncertainty.

![Screenshot at 15:41: 3D visualization of an energy landscape, showing the iterative gradient descent process converging to a low energy state for a given context.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-15-41.png)

**Context:** The video reviews a paper proposing Energy-Based Transformers (EBTs) as a novel approach to machine learning, aiming to enable 'System 2 Thinking' in AI. This concept, borrowed from cognitive science, distinguishes between rapid, intuitive thought (System 1) and slower, deliberate, logical reasoning (System 2). Current large language models (LLMs) are often considered System 1 thinkers, excelling at pattern recognition but struggling with complex, multi-step reasoning or out-of-distribution problems. The paper seeks to bridge this gap by developing models that can 'think' in a System 2 manner using only unsupervised learning, avoiding reliance on explicit human supervision or rule-based rewards.

## Detailed Analysis

This paper explores Energy-Based Transformers (EBTs) as a new paradigm for machine learning models to achieve System 2 Thinking, analogous to human slow, explicit reasoning, solely through unsupervised learning. Unlike traditional models that primarily exhibit System 1 (quick, intuitive) thinking, EBTs are designed to explicitly verify compatibility between inputs and candidate predictions by assigning an energy value, which is then minimized through iterative gradient descent. This formulation allows EBTs to dynamically allocate computation during inference, model uncertainty in continuous state spaces, and verify predictions. Empirically, EBTs demonstrate significant scalability advantages, achieving up to 35% higher scaling rates than Transformer++ models across various axes (data, batch size, depth). They also outperform Diffusion Transformers in image denoising with 99% fewer forward passes. The research suggests that EBTs' thinking capabilities improve generalization, particularly on out-of-distribution data, and offer a promising path for developing more robust and adaptable foundation models without reliance on human supervision or rule-based rewards.

### Core Research Question

- The paper asks if it's possible to generalize System 2 Thinking approaches and develop models that learn to think solely from unsupervised learning, finding the answer to be yes.

### System 1 vs. System 2 Thinking

- System 1 thinking is characterized as quick and intuitive, while System 2 thinking is slow, deliberate, and explicit, involving logical reasoning. Most current machine learning models operate in the System 1 domain, struggling with tasks requiring deeper reasoning or out-of-distribution generalization.

### Energy-Based Transformers (EBTs) Architecture

- EBTs are a new class of Energy-Based Models (EBMs) combined with Transformers, designed to assign an energy value to input-prediction pairs. Lower energy indicates higher compatibility, enabling predictions through gradient descent-based energy minimization until convergence. This architecture supports dynamic computation allocation, modeling uncertainty, and prediction verification.

### EBT Training & Inference

- EBTs are trained to learn an energy landscape where lower energy corresponds to higher compatibility. During inference, EBTs start with an initial random prediction and progressively refine it through iterative energy minimization using gradient descent, allowing models to 'think' longer on harder problems. This process is analogous to System 2 Thinking.

### Scalability & Generalization

- EBTs significantly out-scale the Transformer++ recipe, achieving up to 35% faster scaling rates across data, batch size, and depth. This improved scaling efficiency suggests EBTs are a promising architecture for foundation models. EBTs also demonstrate improved generalization to out-of-distribution data, a key aspect of System 2 Thinking.

### Key Findings & Performance

- EBTs outperform Diffusion Transformers in image denoising with 99% fewer forward passes. The models exhibit strong performance on various language and reasoning benchmarks, including GSM8K, SQuAD, and BigBench Elementary Math QA, indicating robust thinking capabilities.

![Screenshot at 0:00: Paper title 'Energy-Based Transformers are Scalable Learners and Thinkers' with author affiliations.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-00-00.png)
![Screenshot at 1:20: Abstract section with highlighted question: 'Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?'](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-01-20.png)
![Screenshot at 2:30: Handwritten notes differentiating System 1 \(quick, intuitive\) and System 2 \(slow, explicit\) thinking.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-02-30.png)
![Screenshot at 4:55: Diagram illustrating System 1 \(quick, intuitive\) and System 2 \(slow, explicit\) thinking, with System 1 linked to feed-forward transformers and System 2 linked to iterative thinking processes.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-04-55.png)
![Screenshot at 7:58: Comparison of existing autoregressive architectures: AR Transformer, RNN, Diffusion Transformer, and Energy-Based Transformer \(EBT\).](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-07-58.png)
![Screenshot at 10:43: Table outlining three facets of System 2 Thinking \(Dynamic Compute Allocation, Modeling Uncertainty, Prediction Verification\) and how different architectures align with them.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-10-43.png)
![Screenshot at 13:15: Diagram showing EBT for autoregressive modeling, illustrating the iterative thinking process to minimize energy for next token/frame prediction.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-13-15.png)
![Screenshot at 15:41: 3D visualization of an energy landscape, showing the iterative gradient descent process converging to a low energy state for a given context.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-15-41.png)
![Screenshot at 38:45: Graphs comparing EBT and Transformer++ scaling for data, batch size, and depth, showing EBTs out-scaling Transformer++.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-38-45.png)
![Screenshot at 43:12: Table showing EBT performance on various benchmarks \(Pretrain, GSM8K, SQuAD, BB Math QA, BB Dyck\), with lower perplexity indicating better performance.](https://ss.rapidrecap.app/screens/RAEy3JZmIaA/00-43-12.png)