# Sakana.ai: Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

Source: https://www.youtube.com/watch?v=zyTGjCnDaDQ
Recap page: https://rapidrecap.app/video/zyTGjCnDaDQ
Generated: 2026-01-17T13:04:13.158+00:00

---
## Quick Overview

The research paper from Sakana.ai and the University of Oxford demonstrates that dropping positional embeddings (PE) from pretrained Large Language Models (LLMs) allows for efficient, low-cost extrapolation to much longer context lengths, effectively solving the contextual bottleneck that previously required expensive long-context fine-tuning.

**Key Points:**
- Dropping positional embeddings (PE) from pretrained LLMs allows for efficient extension of context length, circumventing the need for expensive long-context fine-tuning.
- The method, dubbed DroP-E (Dropping Positional Embeddings), was tested on the Llama 2 7B model.
- The DroP-E Llama 2 7B model achieved 95% of the original performance on the Haystack benchmark (NAIH), significantly outperforming standard Rope scaling methods.
- The original pre-training cost for the 650 billion token model was $8 million, while the DroP-E fine-tuning cost only $8,000, representing a 99.9% cost reduction for context extension.
- The DroP-E re-calibration phase uses the original short context length (2048 tokens) for a brief adjustment, avoiding instability seen in pure scaling methods.
- The technique works by removing the positional embedding component, which forces the model to learn robust internal representations for positional relationships, leading to superior long-range context understanding.

![Screenshot at 00:14: The title slide of the presentation explicitly names the technique as 'Extending the context of pretrained LLMs by dropping their positional embeddings'.](https://ss.rapidrecap.app/screens/zyTGjCnDaDQ/00-00-14.jpg)

**Context:** The video discusses a new technique called DroP-E (Dropping Positional Embeddings) presented in a paper from Sakana.ai and the University of Oxford, which aims to efficiently extend the context window of existing Large Language Models (LLMs) without the massive computational cost typically associated with fine-tuning for longer contexts. The discussion centers on how positional embeddings limit the ability of models to handle context longer than what they were originally trained on.

## Detailed Analysis

The video introduces research demonstrating that removing positional embeddings (PE) from pretrained LLMs allows them to handle significantly longer context windows efficiently. This technique, called DroP-E, addresses the bottleneck where models struggle to generalize beyond their initial context length without expensive fine-tuning. The researchers applied DroP-E to the Llama 2 7B model. They found that by dropping the PE component, the model could be extrapolated to contexts much longer than its original training length. When tested on the Haystack benchmark (NAIH), the DroP-E model achieved 95% of the performance of the Rope scaling method, which is considered the industry standard. The cost savings are substantial: the fine-tuning cost was reduced from millions to merely $8,000, a 99.9% reduction, using only 0.8% of the original pre-training budget for re-calibration. The method involves a short re-calibration phase using the original context length (2048 tokens) to stabilize the model, after which it performs well on much longer sequences, demonstrating superior ability to maintain semantic connections across vast distances without the inherent bias of positional embeddings.

### Paper Introduction

- Research from Sakana.ai and Oxford
- Technique involves dropping positional embeddings (PE) from pretrained LLMs
- Goal is to extend context length efficiently.

### DroP-E Method Details

- Applied to Llama 2 7B model
- Re-calibration phase uses short context length (2048 tokens) to stabilize the model after PE removal.

### Performance Comparison

- DroP-E achieved 95% of Rope scaling performance on Haystack NAIH benchmark
- Rope scaling showed performance degradation when context was extended.

### Cost Efficiency

- Fine-tuning cost dropped from millions to $8,000 (99.9% reduction)
- Used only 0.8% of the original pre-training budget for re-calibration.

### Theoretical Insight

- Removing PE forces the model to learn internal, robust representations for positional relationships, enabling superior long-range context understanding and avoiding artificial bias.

![Screenshot at 00:26: The speaker explicitly states the acronym for the method being discussed: DroP-E.](https://ss.rapidrecap.app/screens/zyTGjCnDaDQ/00-00-26.jpg)
![Screenshot at 01:19: A comparison is made between the expensive traditional fine-tuning and the cost-effective DroP-E method.](https://ss.rapidrecap.app/screens/zyTGjCnDaDQ/00-01-19.jpg)
![Screenshot at 02:44: The speaker quantifies the cost difference, noting the original pre-training cost was in the millions, while the re-calibration cost was only $8,000.](https://ss.rapidrecap.app/screens/zyTGjCnDaDQ/00-02-44.jpg)
![Screenshot at 04:48: The speaker mentions that the Llama 2 7B model was trained on 4 trillion tokens, providing scale context.](https://ss.rapidrecap.app/screens/zyTGjCnDaDQ/00-04-48.jpg)
![Screenshot at 09:33: The speaker contrasts the success of DroP-E with the previous methods, stating that the no-PE model performed significantly better.](https://ss.rapidrecap.app/screens/zyTGjCnDaDQ/00-09-33.jpg)