Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

Quick Overview

The experiment reveals that large language models (LLMs) fail to correctly answer questions when the context (the story) and the options are presented in reverse order (QOC format), demonstrating a critical limitation in their causal attention mechanism, as models trained linearly (Context-Question-Option) maintained high accuracy, while those trained with context last failed significantly when the order was swapped.

Key Points: Reversing the prompt order from Context-Question-Option (CQO) to Question-Option-Context (QOC) caused a performance drop of nearly 15% across 21 different models, including heavy hitters like GPT-4 and Llama 3. The models failed the 'Lost in the Middle' test when the context was placed last, indicating they struggle to maintain focus on information presented outside of the initial or final parts of the input sequence. When context was at the end (QOC), the models essentially reverted to random guessing, failing to correctly associate the question with the relevant evidence. In contrast, models with the standard CQO format maintained high accuracy regardless of the temporal placement of the evidence within the context. The performance gap between the two formats was approximately 15%, showing that the models are not truly reasoning but are heavily reliant on the order of information presentation. The authors suggest that this failure highlights a fundamental flaw in the causal attention architecture of current LLMs, as they cannot easily look back at context provided later in the prompt sequence.

Context: This video discusses an experimental finding regarding the causal attention mechanism in Large Language Models (LLMs), specifically testing how the order of prompt components—Context, Question, and Options (CQO vs. QOC)—affects model performance on reasoning tasks. The experiment, conducted by researchers from Postech and HJA Lab, involved presenting models with a question, options (A, B, C, D), and a story/evidence, then testing how accuracy changed when the context was moved from the beginning to the end of the prompt.