# Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

Source: https://www.youtube.com/watch?v=mfDDgdpq81o
Recap page: https://rapidrecap.app/video/mfDDgdpq81o
Generated: 2026-01-24T23:03:49.547+00:00

---
## Quick Overview

The experiment reveals that large language models (LLMs) fail to correctly answer questions when the context (the story) and the options are presented in reverse order (QOC format), demonstrating a critical limitation in their causal attention mechanism, as models trained linearly (Context-Question-Option) maintained high accuracy, while those trained with context last failed significantly when the order was swapped.

**Key Points:**
- Reversing the prompt order from Context-Question-Option (CQO) to Question-Option-Context (QOC) caused a performance drop of nearly 15% across 21 different models, including heavy hitters like GPT-4 and Llama 3.
- The models failed the 'Lost in the Middle' test when the context was placed last, indicating they struggle to maintain focus on information presented outside of the initial or final parts of the input sequence.
- When context was at the end (QOC), the models essentially reverted to random guessing, failing to correctly associate the question with the relevant evidence.
- In contrast, models with the standard CQO format maintained high accuracy regardless of the temporal placement of the evidence within the context.
- The performance gap between the two formats was approximately 15%, showing that the models are not truly reasoning but are heavily reliant on the order of information presentation.
- The authors suggest that this failure highlights a fundamental flaw in the causal attention architecture of current LLMs, as they cannot easily look back at context provided later in the prompt sequence.

![Screenshot at 00:48: The moment when the 'Lost in the Prompt Order' experiment is introduced, showing the performance degradation when context is moved to the end of the prompt.](https://ss.rapidrecap.app/screens/mfDDgdpq81o/00-00-48.jpg)

**Context:** This video discusses an experimental finding regarding the causal attention mechanism in Large Language Models (LLMs), specifically testing how the order of prompt components—Context, Question, and Options (CQO vs. QOC)—affects model performance on reasoning tasks. The experiment, conducted by researchers from Postech and HJA Lab, involved presenting models with a question, options (A, B, C, D), and a story/evidence, then testing how accuracy changed when the context was moved from the beginning to the end of the prompt.

## Detailed Analysis

The video details an experiment demonstrating that Large Language Models (LLMs) exhibit a critical failure when the prompt structure is changed from the standard Context-Question-Option (CQO) format to a Question-Option-Context (QOC) format. The researchers tested 21 models, including GPT-4 and Llama 3, on a simple reasoning task (Who stole the apple?) where the correct answer was embedded in the story evidence. When the context was placed last (QOC), performance dropped by nearly 15% on average, proving that the models struggle to attend to information presented late in the sequence, a phenomenon they call 'Lost in the Middle' or, in this specific context, being 'lost in the prompt order.' When the context was presented first (CQO), the models performed well, indicating that the issue is not a lack of information but a flaw in the causal attention mechanism's ability to look backward across the entire sequence. The QOC format forced the model to rely only on the preceding question and options, causing it to ignore the crucial evidence at the end, resulting in an accuracy near zero for that specific format, while the CQO format remained accurate. The researchers conclude that this strongly suggests LLMs are not performing true reasoning based on the full context, but are instead heavily biased by temporal sequencing, proving that order profoundly matters.

### Experimental Setup

- The experiment tests prompt ordering by comparing Context-Question-Option (CQO) against Question-Option-Context (QOC) formats
- The task involved answering a question based on evidence provided in a short story, with options A, B, C, D.

### Results of Prompt Reversal

- Reversing the order (moving context to the end) resulted in an average performance drop of nearly 15% across 21 tested models
- The QOC format led to performance near random guessing (0.0-0.2% accuracy for some metrics).

### Analysis of Failure

- The failure is attributed to the inherent limitations of causal attention, where models struggle to look backward effectively across the entire input sequence
- The decoder-only models are strongly biased by the order of information, failing to connect the question to the evidence when the evidence appears last.

### Implications for AI Architecture

- This reveals a fundamental flaw in current LLM architecture, suggesting that reasoning is not purely based on the data but is highly sensitive to prompt structure
- The gap between the two formats (15%) proves that the models do not possess robust, context-independent reasoning.

![Screenshot at 00:09: The speaker introduces the concept of prompt ordering by stating that the structure feels like a glitch in the matrix.](https://ss.rapidrecap.app/screens/mfDDgdpq81o/00-00-09.jpg)
![Screenshot at 00:48: The specific paper being discussed is identified as 'Lost in the Prompt Order,' which reveals limitations in causal attention.](https://ss.rapidrecap.app/screens/mfDDgdpq81o/00-00-48.jpg)
![Screenshot at 01:24: The speakers mention testing 21 different models, including heavy hitters, to validate the findings.](https://ss.rapidrecap.app/screens/mfDDgdpq81o/00-01-24.jpg)
![Screenshot at 04:33: The speaker outlines the three components of the test: Context, Question, and Options, noting the context is at the bottom in the failed format.](https://ss.rapidrecap.app/screens/mfDDgdpq81o/00-04-33.jpg)
![Screenshot at 05:54: The speaker highlights the simplicity of the test: asking a question and then asking the model to critique a resume, which the model fails when context is last.](https://ss.rapidrecap.app/screens/mfDDgdpq81o/00-05-54.jpg)