# Brumby-14B-Base: The Strongest Attention-Free Base Model

Source: https://www.youtube.com/watch?v=mwcwtftyEDc
Recap page: https://rapidrecap.app/video/mwcwtftyEDc
Generated: 2025-11-11T00:38:59.204+00:00

---
## Quick Overview

The Brumby-14B-Base model demonstrates superior performance by using a novel power retention layer and an RNN structure, achieving state-of-the-art results with only 3,000 steps of fine-tuning and significantly lower computational costs compared to larger models like the 14-billion-parameter base.

**Key Points:**
- Brumby-14B-Base achieves state-of-the-art results despite being significantly smaller than competitors.
- The model utilizes a power retention layer and an RNN structure, which allows for efficient handling of long context.
- Fine-tuning only required 3,000 steps on the base model, making it highly efficient.
- The cost to run the model is estimated to be 50 times cheaper than running a comparable 14-billion-parameter model.
- The power retention layer acts as a direct, effective replacement for the attention layer, preventing the model from running out of memory on long inputs.
- The dual-formulation structure allows the model to process context and perform parallel attention calculations simultaneously.

![Screenshot at 03:43: The speaker highlights the cost argument by contrasting the 14-billion-parameter baseline with the new model's efficiency, calling the cost reduction a genuine shortcut.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-03-43.png)

**Context:** This AI Papers Podcast episode discusses the Brumby-14B-Base model, a large language model (LLM) architecture that challenges the self-attention mechanism dominant in modern LLMs. The speakers focus on how this new architecture, particularly its power retention layer, achieves high performance while drastically cutting down on computational cost and memory requirements compared to traditional, larger Transformer models.

## Detailed Analysis

The discussion centers on the Brumby-14B-Base model, which challenges the established Transformer architecture by replacing the self-attention mechanism with a power retention layer and an RNN structure. This architectural change allows the model to handle long context sequences much more efficiently. The presenters note that the model achieved state-of-the-art results after only 3,000 steps of fine-tuning, a fraction of what is typically required. The cost benefit is substantial: the power retention layer makes inference 50 times cheaper than running a standard 14-billion-parameter model, even for long sequences (1 million tokens). The secret sauce is the dual nature of the structure: it processes tokens sequentially like an RNN while still allowing for parallel computation similar to Transformers, which is essential for modern GPU efficiency. Furthermore, the power retention layer is explicitly stated to be a direct, effective replacement for the attention mechanism, solving the memory bottleneck associated with long context windows. The researchers explicitly state that fine-tuning this model is cheaper and faster than training a large Transformer from scratch, proving that architectural innovation can outperform brute-force scaling.

### Brumby-14B-Base Architecture

- Replaces self-attention with a power retention layer and RNN structure
- Utilizes a dual structure for sequential processing and parallel computation
- Achieves state-of-the-art results with minimal fine-tuning.

### Efficiency and Cost Savings

- Fine-tuned in only 3,000 steps
- Inference cost is 50 times cheaper than comparable 14B models
- Allows processing of 1 million tokens with low cost.

### Technical Mechanics

- Power retention layer acts as a direct replacement for attention
- Solves the memory bottleneck for long context
- Uses an Emi Porson mathematical function and a state matrix.

### Comparison to Established Models

- Outperforms the 14B base model using only 10,000 tokens for context comparison
- Beats larger models in performance while being much cheaper to run.

![Screenshot at 00:01: Podcast intro screen with the title card: "Become A Member Today!"](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-00-01.png)
![Screenshot at 00:30: Speaker discussing the challenge to the basic way modern language models are built.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-00-30.png)
![Screenshot at 00:44: Visual representation of the 14-billion-parameter base model costing significantly more than the new approach.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-00-44.png)
![Screenshot at 01:18: Highlighting the explicit claim that the new model is the strongest attention-free base model.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-01-18.png)
![Screenshot at 02:36: Explaining the novel technique called 'Retraining' used in the new architecture.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-02-36.png)
![Screenshot at 04:50: Speaker detailing the fundamental similarity between attention and power retention layers.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-04-50.png)
![Screenshot at 06:13: Visualizing the RNN structure remembering the beginning of a long document.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-06-13.png)
![Screenshot at 08:17: The speaker points out that the dual-formulation structure is key to its success, contrasting it with pure sequential processing.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-08-17.png)
![Screenshot at 10:00: Speaker summarizing that the efficiency allows for world-class results without massive investment.](https://ss.rapidrecap.app/screens/mwcwtftyEDc/00-10-00.png)