# Context Rot: How Increasing Input Tokens Impacts LLM Performance (Paper Analysis)

Source: https://www.youtube.com/watch?v=hpC4qjWu_aY
Recap page: https://rapidrecap.app/video/hpC4qjWu_aY
Generated: 2025-07-23T12:03:10.029+00:00

---
## Quick Overview

Large Language Models (LLMs) exhibit significant performance degradation, termed 'context rot,' as input token length increases, even on simple tasks like non-lexical retrieval or text replication. This non-uniform performance highlights the critical need for rigorous long-context evaluation and effective context engineering, as simply increasing input size without careful information presentation severely impacts reliability across various models.

**Key Points:**
- LLM performance consistently degrades as input length increases, even on simple tasks like non-lexical retrieval or text replication.
- The degradation is non-uniform, meaning different models and tasks exhibit varying rates of performance decline.
- Lower semantic similarity between the 'needle' (the answer) and the 'question' accelerates performance degradation, making tasks harder for LLMs.
- The presence of 'distractors' (topically related but incorrect information) significantly impacts model performance, leading to further degradation as input length grows.
- Surprisingly, models perform better when the haystack (context) is 'shuffled' (sentences reordered) compared to when it maintains a logical flow, suggesting structural coherence can hinder performance.
- On the LongMemEval benchmark, models show significantly higher performance when given 'focused' prompts (only relevant information) compared to 'full' prompts (all information, including irrelevant content).
- These findings highlight the critical importance of effective 'context engineering' to filter and present only relevant information to LLMs, rather than simply expanding context windows, for reliable real-world performance.

![Screenshot at 2:53: Graph showing LLM performance \(Average Normalized Levenshtein Score\) decreasing as input length \(Tokens\) increases across various models.](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-02-53.png)

**Context:** This video analyzes a research paper titled 'Context Rot: How Increasing Input Tokens Impacts LLM Performance,' authored by researchers from Chroma, a company specializing in vector databases and retrieval engines. The paper challenges the common assumption that Large Language Models (LLMs) process context uniformly, meaning a model should handle the 10,000th token as reliably as the 100th. The speaker introduces the concept of 'context rot,' where LLM performance degrades as the input context length increases, especially with more complex tasks or the presence of irrelevant information. This finding has significant implications for how long-context models are used and how 'context engineering' is approached.

## Detailed Analysis

A paper by Chroma researchers investigates 'context rot' in 18 leading LLMs, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, challenging the assumption that LLMs process context uniformly. Their experiments demonstrate that model performance consistently degrades as input length increases, even on seemingly simple tasks like finding a 'needle' (specific fact) in a 'haystack' (long text). They extend the standard Needle in a Haystack (NIAH) benchmark by varying needle-question similarity, introducing distractors, and altering haystack structure (original vs. shuffled sentences). Results show that lower semantic similarity between needle and question, and the presence of distractors, significantly accelerate performance degradation. Furthermore, models perform better on shuffled haystacks than on logically structured ones, suggesting that structural coherence can surprisingly hinder performance. A more realistic benchmark, LongMemEval, which involves multi-turn conversational question-answering with long, often irrelevant, chat histories, reveals that models perform significantly better when provided with only 'focused' (relevant) parts of the conversation history compared to the 'full' (unfiltered) history. This strongly emphasizes that effective context engineering, rather than merely expanding context windows, is crucial for reliable LLM performance in real-world applications.

### Introduction to Context Rot

- LLMs are presumed to process context uniformly, but in practice, performance degrades as input length increases, especially with complex tasks
- This paper investigates 'context rot' and its implications for long-context models and context engineering
- The findings suggest that smart context engineering, rather than simply stuffing context, leads to better performance.

### Needle in a Haystack (NIAH) Setup

- The classic NIAH task involves placing a 'needle' (fact) within a 'haystack' (long text) and asking the model a question about it
- The original NIAH uses needle-question pairs with lexical matches, making it a narrow capability test
- The report extends NIAH to investigate model behavior in previously underexplored settings, examining effects of semantic vs. lexical matches and varying content.

### Needle-Question Similarity Experiments

- Cosine similarity between needle-question pairs is computed using embeddings to measure performance impact as input length increases
- Models generally perform well at short input lengths, but performance degrades significantly with increasing input length
- Lower similarity between needle and question leads to quicker and more pronounced performance degradation across models.

### Impact of Distractors Experiments

- Distractors are topically related to the needle but do not answer the question, while irrelevant content is unrelated to both needle and question
- Even a single distractor reduces performance relative to the baseline (needle only), and adding four distractors compounds this degradation further
- Models exhibit distinct behaviors in how they deal with ambiguity, with some models showing higher hallucination rates when distractors are present.

### Haystack Structure Experiments

- Haystacks are composed of coherent texts, each with its own logical flow of ideas (original) or sentences are randomly reordered (shuffled)
- Surprisingly, structural coherence consistently hurts model performance, meaning models perform worse when the haystack preserves a logical flow of ideas
- Shuffling the haystack and removing local coherence consistently improves performance, suggesting models are sensitive to the logical flow of context.

### LongMemEval Benchmark

- LongMemEval is a long-context benchmark for conversational question-answering, designed to maintain relevant history for subsequent chats
- It requires models to perform two tasks: retrieval of relevant conversation parts and then reasoning to synthesize an answer
- Models show significantly higher performance on 'focused' prompts (containing only relevant parts) compared to 'full' prompts (containing all chat history, including irrelevant content and distractors).

### Conclusion

- LLMs do not maintain consistent performance across input lengths, even on simple tasks like non-lexical retrieval or text replication
- There is increasing non-uniformity in performance as input length grows, highlighting the need for more rigorous long-context evaluation
- Relevant information presence in a model's context is not all that matters; how that information is presented is crucial, making effective context engineering essential for reliable performance.

![Screenshot at 0:00: Speaker presenting the title slide 'Context Rot: How Increasing Input Tokens Impacts LLM Performance'](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-00-00.png)
![Screenshot at 2:53: Graph titled 'Repeated Words - Performance by Input Length \(Tokens\)' showing various LLM models' performance decreasing with increasing input length.](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-02-53.png)
![Screenshot at 5:10: Diagram illustrating the 'Needle in a Haystack' \(NIAH\) setup with a 'Question', 'Needle' \(the answer\), and 'Haystack' \(the context\).](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-05-10.png)
![Screenshot at 1:19: 58:Diagram showing 'Impact of Distractors' with three conditions: 'No Distractors', '1 Distractor', and 'All 4 Distractors' within the haystack.](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-01-19.png)
![Screenshot at 1:47: 40:Bar charts comparing 'Average Accuracy' for 'Original' vs. 'Shuffled' haystacks, showing higher accuracy for shuffled contexts.](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-01-47.png)
![Screenshot at 2:08: 46:Diagrams illustrating 'knowledge-update', 'temporal-reasoning', and 'multi-session' tasks within the LongMemEval benchmark, showing user and assistant turns.](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-02-08.png)
![Screenshot at 2:11: 25:Bar chart showing 'LongMemEval Overall Performance - Claude Family', comparing 'Focused' \(red\) vs. 'Full' \(blue\) prompts across different Claude models, with 'Focused' consistently outperforming 'Full'.](https://ss.rapidrecap.app/screens/hpC4qjWu_aY/00-02-11.png)