# Rich Sutton, The OaK Architecture: A Vision of SuperIntelligence from Experience - RLC 2025

Source: https://www.youtube.com/watch?v=gEbbGyNkR2U
Recap page: https://rapidrecap.app/video/gEbbGyNkR2U
Generated: 2025-08-23T04:50:28.668+00:00

---
## Quick Overview

Rich Sutton presents the OAK architecture as a vision for achieving superintelligence through experience, emphasizing that true AI progress lies in improving reinforcement learning algorithms rather than relying on non-experiential methods like large language models. OAK is designed to be domain-general, experiential, and open-ended in sophistication, learning entirely at runtime from the complex, big world rather than through pre-programmed design-time knowledge. The architecture aims to enable agents to discover high-level abstractions and reasoning capabilities through self-generated sub-problems, mirroring natural play and learning processes.

**Key Points:**
- Rich Sutton proposes the OAK architecture as a vision for superintelligence driven entirely by experience, emphasizing that the path to AI lies in improving reinforcement learning algorithms.
- OAK is designed to be domain-general, experiential (learning only at runtime), and open-ended in sophistication, enabling agents to discover abstractions in complex, 'big worlds'.
- The 'big world' hypothesis posits that the world is vastly larger and more complex than any agent, necessitating runtime learning and adaptation due to non-stationarity and approximation.
- A core innovation of OAK is the generation of self-posed sub-problems, inspired by natural play and curiosity, to drive the learning of new features and higher-level concepts.
- Sutton highlights that while planning with temporally extended behaviors (options) is understood, reliable continual deep learning and robust feature generation remain critical open challenges for the architecture.
- The architecture aims to provide plausible mechanistic answers to fundamental AI questions, including how high-level knowledge is learned from low-level experience and the origin of concepts.

**Context:** Rich Sutton, a prominent researcher in reinforcement learning, delivered this talk at the RLC 2025 conference. He has dedicated half a century to developing reinforcement learning algorithms and is known for his 'bitter lesson' and 'reward hypothesis' ideas. This presentation introduces his OAK (Options, Knowledge) architecture, a new vision for AI agent design aimed at achieving superintelligence through experience, contrasting it with approaches that rely heavily on pre-programmed knowledge or non-experiential methods like large language models.

## Detailed Analysis

Rich Sutton introduces the OAK (Options, Knowledge) architecture as a path to superintelligence, asserting that the future of AI hinges on advancing reinforcement learning (RL) algorithms, not on non-experiential approaches like LLMs. He posits that AI is a grand quest comparable to understanding the origin of life, and the greatest advances are still ahead. Sutton critiques current algorithms as crude and highlights the need for better learning algorithms. The OAK architecture is built on three core design principles: domain generality, experiential learning (all learning occurs at runtime, not from special training phases), and open-ended sophistication for forming any necessary concepts. He distinguishes between design time (where domain knowledge can be embedded) and runtime (where interaction and learning occur), advocating for the latter to be the primary locus of activity due to the immense complexity of the 'big world.' The 'big world' hypothesis suggests the world is vastly larger and more complex than any agent, leading to approximate, non-stationary, and learned behaviors. OAK utilizes 'options' (pairs of policy and termination condition) and learns knowledge about following these options to build high-level transition models for planning. Key to OAK is the generation of self-posed sub-problems, inspired by natural play and curiosity, which drive the discovery of new features and abstractions. These sub-problems, framed as 'reward-respecting feature attainment,' allow agents to learn to achieve specific features without sacrificing overall reward. The architecture involves several parallel runtime steps: learning main reward maximization, generating new state features, creating sub-problems for highly-ranked features, learning options (solutions to sub-problems), learning models of options, and planning with these models. While many components are conceptually understood (blue checkmarks), reliable continual deep learning and robust feature generation remain open challenges (red checkmarks), though promising avenues like continual backprop and meta-learning are being explored. Sutton emphasizes that planning with option models is crucial for adapting to the dynamic 'big world,' and while traditional planning uses value iteration, OAK extends this to temporally extended behaviors (options). He concludes by framing OAK as a plausible mechanistic answer to fundamental AI questions about learning high-level knowledge from experience, the origin of concepts, the nature of reasoning, the purpose of play, and perception without external labels.

### Introduction to OAK Architecture

- Vision for Superintelligence from Experience
- Reinforcement Learning as the Path Forward
- Critique of Current AI Algorithms

### Core Design Principles of OAK

- Domain General
- Experiential Learning (Runtime Focus)
- Open-ended Sophistication

### The 'Big World' Hypothesis

- Agent vs. World Complexity
- Runtime Learning Imperative
- Non-stationarity and Approximation

### OAK Architecture Mechanics

- Options and Knowledge
- Sub-problems and Feature Attainment
- Learning Models and Planning with Options

### Runtime Processes of OAK

- Parallel Steps
- Key Challenges (Continual Learning, Feature Generation)
- Potential Solutions (Continual Backprop, Meta-learning)

### Planning in OAK

- Adapting to the Big World
- Value Iteration with Options
- Function Approximation

### Fundamental AI Questions Addressed by OAK

- Learning High-Level Knowledge
- Origin of Concepts
- Nature of Reasoning and Play
- Perception without Labels