# Long context windows for LLMs: 1 million tokens and beyond | Lex Fridman Podcast Source: https://www.youtube.com/watch?v=6iq0QXRRyso Recap page: https://rapidrecap.app/video/6iq0QXRRyso Generated: 2026-02-04T05:32:45.381+00:00 --- ## Quick Overview Recursive Language Models (RLMs) offer a general inference paradigm that treats long prompts as part of an external environment, allowing LLMs to programmatically examine, decompose, and recursively call itself over snippets of the prompt, successfully handling inputs up to two orders of magnitude beyond standard model context windows with comparable cost, as demonstrated by RLM-Queen-8B outperforming the underlying model by 28.3% on average. **Key Points:** - Recursive Language Models (RLMs) treat long prompts as an external environment, enabling programmatic examination, decomposition, and recursive self-calling over prompt snippets. - RLMs successfully process inputs up to two orders of magnitude beyond typical model context windows, maintaining comparable computational cost. - The RLM-Queen-8B model, pre-trained on 32k context length, outperforms the underlying model by 28.3% on average across various long-context tasks. - The method avoids the computational expense of brute-forcing 1 million token context windows by selectively attending to relevant context chunks, contrasting with simpler context compression techniques. - The researchers found that the RLM approach, when pre-trained on 8k context, achieved performance comparable to models with 32k context length, suggesting efficiency gains. - The primary benefit of this approach is achieving better performance on long-context tasks without the prohibitive computational cost associated with naively increasing context length across the entire input sequence. ![Screenshot at 00:04: 24:The host, Lex Fridman, listens intently while the guest explains the concept of Recursion in Language Models, contrasting it with brute-force context expansion.](https://ss.rapidrecap.app/screens/6iq0QXRRyso/00-00-04.jpg) **Context:** This segment of the Lex Fridman Podcast features a discussion about advancements in Large Language Models (LLMs), specifically focusing on the concept of Recursive Language Models (RLMs). The conversation centers on overcoming the limitation of fixed, short context windows in LLMs by introducing a new inference paradigm that allows the model to manage and process arbitrarily long prompts efficiently. The participants discuss the technical approach of RLMs, which involves programmatic decomposition and recursive self-reference rather than simple context window expansion. ## Detailed Analysis The discussion centers on Recursive Language Models (RLMs) as a solution for handling arbitrarily long prompts in LLMs, addressing the limitations of inference-time context scaling. The RLMs framework treats the long prompt as part of an external environment, allowing the LLM to programmatically examine, decompose, and recursively call itself over smaller snippets of the prompt. This method successfully handles inputs up to two orders of magnitude larger than standard context windows while maintaining comparable computational cost. The guest mentions their model, RLM-Queen-8B, which, when pre-trained on 8k context and then fine-tuned with longer contexts, outperforms the underlying model by 28.3% on average, approaching the quality of GPT-3.5 on 32k context tasks. A key distinction is made between RLMs and other compression methods like sparse attention or sliding window attention, as RLMs allow the model to decide what context to focus on, rather than relying on fixed or sparse mechanisms. The speaker notes that while a brute-force approach like training on 1 million tokens is expensive, the RLM approach allows for more efficient scaling and better performance on long-context reasoning tasks. The concept is likened to an agent managing context, which contrasts with the 'brute force expensive thing' of simply making context windows huge. ### Recursive Language Models (RLMs) - RLMs treat long prompts as an external environment - LLM programmatically examines, decomposes, and recursively calls itself over snippets - Successfully handles inputs up to two orders of magnitude beyond model context windows ### Performance and Efficiency - RLM-Queen-8B outperforms the underlying model by 28.3% on average - Achieves performance comparable to models trained on 32k context, even when starting with 8k pre-training - Avoids the high computational cost of brute-forcing massive context lengths ### Comparison to Other Methods - Contrasts with sparse attention or sliding window attention where context is fixed or selectively masked - RLMs offer a novel way to manage context by making the model itself the agent controlling context flow ### Future Outlook - Discusses potential for future models to be smarter about context management, moving away from simple context window size increases ![Screenshot at 00:03: 00:Lex Fridman listening intently while the guest explains the concept of context management in LLMs.](https://ss.rapidrecap.app/screens/6iq0QXRRyso/00-00-03.jpg) ![Screenshot at 00:09: 00:The guest elaborates on the difference between accepting context as a simple data problem versus using recursive strategies.](https://ss.rapidrecap.app/screens/6iq0QXRRyso/00-00-09.jpg) ![Screenshot at 00:24: 00:The guest gestures while explaining how context management—like the Mamba architecture—is better suited for long sequences than standard Transformers.](https://ss.rapidrecap.app/screens/6iq0QXRRyso/00-00-24.jpg) ![Screenshot at 00:42: 00:The guest details the necessity of finding good ratios for attention layers in long context tasks.](https://ss.rapidrecap.app/screens/6iq0QXRRyso/00-00-42.jpg) ![Screenshot at 00:42: 00:The paper slide detailing 'Recursive Language Models' and showing performance graphs comparing GPT-3 and RLM\(GPT-3\) performance across increasing input context lengths.](https://ss.rapidrecap.app/screens/6iq0QXRRyso/00-00-42.jpg)