Bubble or No Bubble, AI Keeps Progressing (ft. Relentless Learning + Introspection)

Quick Overview

The video argues that despite the current AI hype cycle potentially being a bubble, the underlying progress in AI, especially demonstrated by architectures like Nested Learning (NL) and the HOPE model, shows tangible, non-hyped advancements in continual learning and introspection capabilities that will drive future progress regardless of short-term market sentiment.

Key Points: The speaker argues that current AI progress, exemplified by models using Nested Learning (NL) and HOPE, is real and distinct from speculative hype, suggesting a plateau or bubble in certain areas does not negate fundamental advancement. The HOPE architecture, a variant of the Titans architecture, utilizes NL principles to prioritize long-term memory modules that learn based on surprise, achieving better performance than existing models on benchmarks like perplexity and common-sense reasoning. The video contrasts the slow progress in the 60s-80s with the current rapid development, noting that models like GPT-4 and Gemini 3, while highly capable, still lack true introspection and continual learning abilities that HOPE aims to address. Anthropic's research on Claude models demonstrated that AI can be prompted to introspect, revealing an "all caps" vector associated with concepts like 'LOUD' or 'SHOUTING', even when the model is not explicitly instructed to reveal internal states. The speaker highlights the importance of continual learning, suggesting that models should learn from practice (like new code or specifications) rather than just being pre-trained, which is the focus of the HOPE architecture. A comparison chart shows HOPE outperforming competitors like Titans, Samba, and the baseline Transformer across language modeling (perplexity) and common-sense reasoning tasks at the 1.3B parameter scale. The speaker concludes that the demonstrable ability of models to introspect and continually learn, even if imperfectly, suggests a fundamental shift in AI capability beyond current market valuations.