# New AI Paradigm?! Energy-Based Transformers Explained

Source: https://www.youtube.com/watch?v=LUQkWzjv2RM
Recap page: https://rapidrecap.app/video/LUQkWzjv2RM
Generated: 2025-09-06T00:30:39.997+00:00

---
## Quick Overview

Energy-Based Transformers (EBTs) are a new AI paradigm that outperforms standard Transformer++ models in reasoning, out-of-distribution generalization, and certain benchmarks, despite being computationally more expensive and requiring more memory.

**Key Points:**
- EBTs demonstrate superior performance over Transformer++ models in reasoning, out-of-distribution generalization, and three out of four benchmarks.
- EBTs achieve better denoising quality in out-of-distribution scenarios compared to Diffusion Transformers (DiTs), requiring only 1% of the forward passes for comparable or better PSNR.
- EBTs can be implemented with transformers and scale faster during pretraining than Transformer++ models.
- The 'thinking' capability of EBTs allows them to adapt to more complex problems and generalize better to out-of-distribution data.
- EBTs utilize an iterative energy minimization process to refine predictions, allowing for more flexible computation and better uncertainty estimation.
- While EBTs are computationally more expensive and use more memory, their improved performance and scalability suggest they will outperform Transformer++ models at foundation model scale.

![Screenshot at 00:00: A bar chart displays the top 3 Large Language Models \(LLMs\) as of August 12, 2025, with GPT-5 \(high\) in reasoning ranked first, Gemini 2.5 Pro \(reasoning\) second, and Claude Opus 4.1 \(reasoning\) third.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-00.png)

**Context:** This video introduces Energy-Based Transformers (EBTs), a new type of AI model that aims to improve reasoning capabilities in large language models. It contrasts EBTs with existing Transformer models, highlighting their unique approach to learning and prediction through an iterative energy minimization process. The video discusses the performance benefits of EBTs, particularly in terms of reasoning and out-of-distribution generalization, while also acknowledging their computational costs.

## Detailed Analysis

The video introduces Energy-Based Transformers (EBTs) as a new AI paradigm that shows promise in improving reasoning capabilities in large language models. Unlike standard Transformer++ models which have fixed computational budgets, EBTs can dynamically adjust their computation by iterating through an optimization process, allowing them to estimate uncertainty and verify predictions. The research presented demonstrates that EBTs outperform Transformer++ models in several key areas, including reasoning tasks, out-of-distribution (OOD) generalization, and scaling efficiency across data, batch size, and depth. Specifically, EBTs achieve better denoising quality in OOD scenarios than diffusion models, requiring significantly fewer forward passes for comparable results. While EBTs are computationally more expensive and require more memory than standard transformers, their ability to scale efficiently and generalize better suggests they will likely outperform existing models at foundation model scale. The 'thinking' process of EBTs, which involves iterative refinement of predictions based on an energy landscape, is crucial for their robust generalization capabilities, especially on OOD data. The research indicates a roughly linear trend between the degree of out-of-distribution data and the performance improvements gained from EBTs' thinking process.

### Introduction to EBTs

- EBTs are a new class of Energy-Based Models (EBMS) that assign an energy value to input-prediction pairs, enabling predictions through gradient-based energy minimization.

### EBTs vs. Transformer++

- EBTs outperform Transformer++ in scaling for data, batch size, and depth, indicating improved data efficiency and potential for generalization.

### EBTs and Reasoning

- Improved depth scaling in EBTs shows promise for reasoning tasks, where depth is crucial.

### EBT Performance Gains

- EBTs scale faster than Transformer++ during pretraining and achieve higher scaling rates, suggesting they will outperform Transformer++ models at foundation model scale.

### OOD Performance

- EBTs generalize better out-of-distribution (OOD) than Transformer++ due to their System 2 thinking capabilities and scaling thinking capabilities during training.

### Thinking Process

- EBTs learn uncertainty across text tokens without explicit supervision, with simpler tokens having lower uncertainty and harder tokens having higher uncertainty.

### Limitations

- EBTs are computationally more expensive and require more memory, and their training and inference processes are more complex.

### Future Potential

- EBTs show promising scalability and performance improvements, suggesting they could be a new paradigm for scaling both learning and thinking capabilities of models.

![Screenshot at 00:00: A bar chart shows the top 3 LLMs as of August 12, 2025, with GPT-5 ranked first, Gemini 2.5 Pro second, and Claude Opus 4.1 third.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-00.png)
![Screenshot at 00:03: A screenshot of ChatGPT's interface, showing a mathematical problem and its step-by-step solution.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-03.png)
![Screenshot at 00:08: Two robots present a mathematical problem and its solution, indicating higher chance of correctness with a detailed calculation and higher chance of incorrectness with a single numerical answer.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-08.png)
![Screenshot at 00:13: A "reasoning yapping levels" gauge shows a needle pointing to "Severe."](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-13.png)
![Screenshot at 00:17: A manga panel depicts a stern-looking man towering over a frightened child.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-17.png)
![Screenshot at 00:19: A slide explains "Step Three: Deciding the Multiplication Order" with Kermit the Frog present.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-19.png)
![Screenshot at 00:20: A slide explains "Step Seven: Reflecting on the Process" with Kermit the Frog present.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-20.png)
![Screenshot at 00:24: A diagram illustrates the "PRM Training Pipeline \(Micromanaging in Training\)".](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-24.png)
![Screenshot at 00:29: A "Reward Signal Accuracy Scale" shows a gradient from -1 \(Incorrect\) to 1 \(Correct\).](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-29.png)
![Screenshot at 00:34: A code editor displays Python code with a calculator interface on the right showing a mathematical expression being typed in, with suggestions appearing in the code editor.](https://ss.rapidrecap.app/screens/LUQkWzjv2RM/00-00-34.png)