# Bubble or No Bubble, AI Keeps Progressing (ft. Relentless Learning + Introspection) Source: https://www.youtube.com/watch?v=Dl3Olh29_nY Recap page: https://rapidrecap.app/video/Dl3Olh29_nY Generated: 2025-11-10T16:09:50.446+00:00 --- ## Quick Overview The video argues that despite the current AI hype cycle potentially being a bubble, the underlying progress in AI, especially demonstrated by architectures like Nested Learning (NL) and the HOPE model, shows tangible, non-hyped advancements in continual learning and introspection capabilities that will drive future progress regardless of short-term market sentiment. **Key Points:** - The speaker argues that current AI progress, exemplified by models using Nested Learning (NL) and HOPE, is real and distinct from speculative hype, suggesting a plateau or bubble in certain areas does not negate fundamental advancement. - The HOPE architecture, a variant of the Titans architecture, utilizes NL principles to prioritize long-term memory modules that learn based on surprise, achieving better performance than existing models on benchmarks like perplexity and common-sense reasoning. - The video contrasts the slow progress in the 60s-80s with the current rapid development, noting that models like GPT-4 and Gemini 3, while highly capable, still lack true introspection and continual learning abilities that HOPE aims to address. - Anthropic's research on Claude models demonstrated that AI can be prompted to introspect, revealing an "all caps" vector associated with concepts like 'LOUD' or 'SHOUTING', even when the model is not explicitly instructed to reveal internal states. - The speaker highlights the importance of continual learning, suggesting that models should learn from practice (like new code or specifications) rather than just being pre-trained, which is the focus of the HOPE architecture. - A comparison chart shows HOPE outperforming competitors like Titans, Samba, and the baseline Transformer across language modeling (perplexity) and common-sense reasoning tasks at the 1.3B parameter scale. - The speaker concludes that the demonstrable ability of models to introspect and continually learn, even if imperfectly, suggests a fundamental shift in AI capability beyond current market valuations. ![Screenshot at 00:22: The video transitions from a Google search about the AI bubble to the first page of the "Nested Learning: The Illusion of Deep Learning Architectures" paper, featuring authors from Google Research and an abstract detailing the new learning paradigm.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-00-22.png) **Context:** The video explores the current state of Large Language Models (LLMs) by examining recent research papers, specifically focusing on the "Nested Learning" paradigm introduced by Google researchers and the "HOPE" architecture developed by Anthropic. The context is set against the backdrop of public discourse surrounding an "AI bubble," contrasting short-term market hype with evidence of genuine, underlying technical progress in areas like continual learning and model introspection. ## Detailed Analysis The video argues that while the AI market might be experiencing a bubble, fundamental progress continues, focusing on two key areas: continual learning and model introspection. The speaker first discusses the Google research on Nested Learning (NL), which proposes a new learning paradigm that coherently represents models as nested optimization problems, allowing models to learn continuously without forgetting prior knowledge, contrasting this with previous models that struggled with catastrophic forgetting. The speaker specifically mentions the authors from Google Research and the title of their paper, "Nested Learning: The Illusion of Deep Learning Architectures" (00:22). The speaker then pivots to Anthropic's research on Claude models, which demonstrated evidence of introspection by showing the model could detect an "injected thought" vector, such as the concept 'LOUD' or 'SHOUTING', when probed (07:37). This ability, though currently unreliable, suggests models can monitor their own internal states. The speaker compares the performance of HOPE (a model using NL) against other architectures like Titans, Samba, and Transformers, showing HOPE achieving lower perplexity and higher accuracy on language modeling and common-sense reasoning tasks (10:46). The speaker concludes that this tangible progress in continual learning and introspection suggests AI development is proceeding on solid technical ground, regardless of short-term market fluctuations or potential bubbles. ### AI Progress vs. Hype - Progress in continual learning (Nested Learning/HOPE) and introspection (Anthropic's Claude research) is real, contrasting with market hype and the plateauing of progress in older models like GPT-3 (00:03-00:22). - Fundamental Architectural Advancements: Google's NL paradigm structures models as nested optimization problems to enable continual learning, while Anthropic's HOPE model is a self-referential architecture designed to avoid catastrophic forgetting (01:05-01:35). - Model Introspection Evidence: Anthropic found evidence that Claude models can detect injected concepts (like the 'all caps' vector associated with 'LOUD') by monitoring internal activations, showing a degree of self-awareness (07:37-08:34). - Performance Benchmarks: A chart compares HOPE against Titans, Samba, and Transformer models, showing HOPE achieving superior performance on common-sense reasoning and lower perplexity on language modeling tasks (10:46-10:50). - Future Implications: The speaker suggests that models capable of introspection and continual learning will be crucial for future AI safety and development, contrasting this deep progress with the current focus on superficial advancements (08:38-09:06). ![Screenshot at 00:00: Google search results displaying multiple news articles discussing the "AI bubble" from sources like Citywire and Financial Times.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-00-00.png) ![Screenshot at 00:22: The first page of the Google research paper titled "Nested Learning: The Illusion of Deep Learning Architectures" by Ali Behrouz et al.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-00-22.png) ![Screenshot at 01:20: A cartoon image illustrating an extreme scenario on a chessboard involving an orca, a beluga whale, a polar bear, and a kayaker, used as a visual metaphor.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-01-20.png) ![Screenshot at 01:44: A table comparing the performance of HOPE against various models \(Transformer++, BetaNet, DeltaNet, TTT, Samba, Titans\) across multiple language modeling benchmarks \(Wiki, LMB, etc.\) at 760M and 1.3B parameters.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-01-44.png) ![Screenshot at 07:37: A diagram from the Anthropic research showing the methodology for extracting an 'all caps' vector by comparing internal activations between two prompts.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-07-37.png) ![Screenshot at 07:58: A side-by-side comparison showing the Default Response \(no detection\) versus the Injecting the 'all caps' vector \(detection\) output from the Claude model.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-07-58.png) ![Screenshot at 10:00: A screenshot of an article discussing the gap between how people use AI and its current capabilities, highlighting text about the "immense" gap.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-10-00.png) ![Screenshot at 10:36: A graphic listing four "Featured Challenges" from a platform, including "Indirect Prompt Injection" and "Proving Ground," with associated award amounts and progress status.](https://ss.rapidrecap.app/screens/Dl3Olh29_nY/00-10-36.png)