# RIFT: Reordered Instruction Following To Evaluate Instruction Following in Singular Multistep Prompt

Source: https://www.youtube.com/watch?v=WW3kdMuEnf0
Recap page: https://rapidrecap.app/video/WW3kdMuEnf0
Generated: 2026-02-02T22:05:02.953+00:00

---
## Quick Overview

The RIFT paper demonstrates that large language models (LLMs) struggle significantly with multi-step instruction following when the steps are reordered, leading to a substantial drop in accuracy (from 85% baseline to 13.9% in one test) due to a failure to maintain a global state, suggesting structural flaws rather than mere capacity issues in current models.

**Key Points:**
- LLMs demonstrated a significant drop in performance when instruction order was reordered (RIFT test), with accuracy plummeting from an 85% baseline to 13.9% for the 120B parameter model.
- The failure is attributed to an architectural flaw, specifically the inability to maintain a global state across steps, rather than a capacity issue.
- The 120B parameter model performed slightly better (85% baseline accuracy) on the linear instruction set compared to the 30B parameter model's performance on the same test.
- When tested with jumping prompts (non-sequential instruction following), the 120B model's accuracy dropped significantly, while the smaller 30B model performed nearly at random (0% accuracy).
- The research suggests that while larger models are better at reasoning and data retention, they remain fragile against structural challenges like context window shifts or non-linear instructions.
- The failure mode observed in the jumping prompt test was characterized by the model either stopping early or aggressively moving to the end of the sequence, indicating a breakdown in tracking progress.
- The paper concludes that the current architectural approach of relying heavily on positional bias (what is physically nearby in the text) is a major limitation for complex, multi-step reasoning.

![Screenshot at 00:03: The video displays a title card mentioning the paper "RIFT: Reordered Instruction Following To Evaluate Instruction Following in Singular Multistep Prompt," setting the context for evaluating LLMs on instruction adherence.](https://ss.rapidrecap.app/screens/WW3kdMuEnf0/00-00-03.jpg)

**Context:** The video discusses findings from the research paper titled "RIFT: Reordered Instruction Following To Evaluate Instruction Following in Singular Multistep Prompt," which investigates the robustness of large language models (LLMs) when the order of instructions within a prompt is intentionally scrambled. The research, conducted by Emory University researchers including Andrew Jaffe, Noah Risen, and Genio DeTroy, highlights a critical weakness in how current state-of-the-art models handle sequential dependencies, contrasting their performance on linear versus non-linear instruction sets.

## Detailed Analysis

The discussion centers on the RIFT paper, which examines how large language models (LLMs) handle instructions when their order is deliberately changed, contrasting this with standard linear instruction following. The researchers tested models like the 120B parameter model and the 30B parameter model. On a baseline test where instructions are given sequentially (linear mode), the 120B model achieved 85% accuracy. However, when the same instructions were reordered (jumping prompts), the 120B model's accuracy dropped drastically to 13.9%, while the smaller 30B model performed near randomly (0% accuracy). This severe performance degradation suggests the issue is not merely capacity but an architectural vulnerability—the inability to maintain a global state or track progress through a sequence when context is disrupted. The paper suggests models rely too heavily on positional bias (what text is physically close). Even the larger models, despite superior reasoning and data retention, fail when asked to follow instructions out of order, leading to poor performance on tasks like navigating a book or executing complex plans. The conclusion is that current LLM architectures have a fundamental flaw in handling non-linear procedural tasks, which is a significant limitation for building reliable AI agents.

### RIFT Paper Overview

- Paper name is RIFT (Reordered Instruction Following To Evaluate Instruction Following in Singular Multistep Prompt)
- Researchers from Emory University
- Focus on testing LLMs' ability to follow instructions regardless of their order.

### Linear vs. Jumping Prompts

- Baseline accuracy (linear) for 120B model was 85%
- Accuracy dropped to 13.9% with reordered (jumping) prompts for the 120B model
- 30B model accuracy dropped to 0% on jumping prompts.

### Nature of Failure

- The failure is an architectural flaw (inability to maintain global state), not a capacity issue
- Models fail to maintain context when instructions are not sequential.

### Specific Failures Observed

- Models often output the answer immediately if they know it, or stop processing early, indicating they lose track of where they are in the instruction sequence.

### Implications for AI Agents

- The reliance on positional bias (proximity in text) is a critical limitation
- This structural flaw means models cannot reliably perform real-world workflows that require maintaining state across non-sequential steps.

![Screenshot at 00:00: Introductory visual featuring two podcasters under an overlay graphic suggesting an audio frequency wave, with text urging viewers to 'BECOME A MEMBER TODAY!'](https://ss.rapidrecap.app/screens/WW3kdMuEnf0/00-00-00.jpg)
![Screenshot at 00:27: The speaker discusses the tension between marketing claims and performance reality regarding large language models.](https://ss.rapidrecap.app/screens/WW3kdMuEnf0/00-00-27.jpg)
![Screenshot at 00:40: The speaker mentions the performance drop for the 120B model when tested with jumping prompts, falling from 85% to 13.9%.](https://ss.rapidrecap.app/screens/WW3kdMuEnf0/00-00-40.jpg)
![Screenshot at 01:57: The speaker explains that the smaller models used jeopardy questions to test factual recall, which is easier than procedural tasks.](https://ss.rapidrecap.app/screens/WW3kdMuEnf0/00-01-57.jpg)
![Screenshot at 03:34: The speaker details the parameter sizes tested: 120B and 30B versions of the Qwen models, noting both parameter and context window versions.](https://ss.rapidrecap.app/screens/WW3kdMuEnf0/00-03-34.jpg)