# Continual Learning: The big challenge for AI - How to learn new things quickly | Lex Fridman Podcast

Source: https://www.youtube.com/watch?v=ohRQpK_CKG4
Recap page: https://rapidrecap.app/video/ohRQpK_CKG4
Generated: 2026-02-06T21:05:55.022+00:00

---
## Quick Overview

The primary challenge in achieving true Artificial General Intelligence (AGI) lies in developing Continual Learning (CL) systems that can permanently accumulate knowledge without catastrophic forgetting, contrasting sharply with current In-Context Learning (ICL) methods that are transient and context-window limited.

**Key Points:**
- Continual Learning (CL) involves gradient-based updates that permanently change a model's weights to accumulate knowledge from streaming data, handling non-stationary data distributions.
- In-Context Learning (ICL) relies on a frozen model adapting by conditioning predictions on examples provided in the input prompt; knowledge is transient and lost when the context window clears.
- The cost of training/fine-tuning for CL is high during the learning phase, whereas ICL has low setup cost (inference only), but high per-token cost if the context is long.
- The capacity for CL is theoretically unbounded (limited by network capacity), while ICL capacity is strictly limited by the model's context window size.
- A key milestone for AGI, as discussed, is when AI can learn like humans, adapting its weights based on new information, which ICL currently cannot do.
- The expense of updating weight matrices for every individual user (personalization) is a significant economic barrier to scaling CL across a large user base.
- The paper "S-LoRA: Serving Thousands of Concurrent LoRA Adapters" suggests a scalable approach to serving many task-specific fine-tuned models (like LoRA adapters) concurrently.

![Screenshot at 00:03: Lex Fridman sitting at his desk, leaning on his hand near the microphone, reflecting on the complexity of continual learning challenges.](https://ss.rapidrecap.app/screens/ohRQpK_CKG4/00-00-03.jpg)

**Context:** This segment of the Lex Fridman Podcast features a discussion comparing two core concepts in large language model development: Continual Learning (CL) and In-Context Learning (ICL). Lex Fridman and his guest, likely an AI researcher given the technical nature of the discussion, analyze the fundamental differences between these paradigms regarding knowledge persistence, computational cost, and capacity, framing CL as the necessary path toward AGI while acknowledging the practical limitations of current ICL approaches.

## Detailed Analysis

The discussion centers on the distinction between Continual Learning (CL) and In-Context Learning (ICL) as they relate to advancing AI toward AGI. CL is defined as a process where a model continuously learns from a stream of data, updating its parameters (weights) permanently to accumulate knowledge, which is essential for handling non-stationary data. In contrast, ICL keeps the model weights frozen and relies solely on conditioning the output based on examples within the immediate, transient context window. The implications are significant: CL requires high compute cost for training/fine-tuning but offers theoretically unbounded capacity, whereas ICL has low inference cost but is strictly limited by the context window size. The guest emphasizes that true generalized intelligence requires the ability to update weights dynamically, which ICL lacks. A major hurdle for scaling CL is the economic cost of updating weights for every individual user (personalization), making it expensive compared to caching ICL outputs. The conversation touches upon the paper "S-LoRA: Serving Thousands of Concurrent LoRA Adapters" (04:48), which addresses scaling the serving of many task-specific fine-tuned models, suggesting methods to mitigate high training costs by efficiently managing adapters, though the core challenge of permanent, rapid learning remains central to achieving AGI.

### Continual Learning vs. In-Context Learning

- CL involves permanent weight updates and handles non-stationary data; ICL uses frozen weights, relies on transient context, and handles stationary data.

### Cost and Capacity

- CL has high initial training cost but unbounded capacity; ICL has low inference cost but capacity is strictly limited by the context window size.

### AGI Requirement

- True AGI requires models that can continuously adapt weights (CL), which ICL cannot achieve, leading to the problem of catastrophic forgetting.

### Practical Challenges

- The cost of personalized, continual weight updates for every user is economically prohibitive currently, contrasting with the cheaper, context-fed approach of ICL.

### Related Research

- The S-LoRA paper addresses serving thousands of concurrent LoRA adapters efficiently, improving throughput by up to 4x, offering a pathway to scale customized fine-tuning.

![Screenshot at 00:03: Lex Fridman sitting at his desk, leaning on his hand near the microphone, reflecting on the complexity of continual learning challenges.](https://ss.rapidrecap.app/screens/ohRQpK_CKG4/00-00-03.jpg)
![Screenshot at 00:05: Lex Fridman appears thoughtful while discussing the concepts, resting his head on his hand.](https://ss.rapidrecap.app/screens/ohRQpK_CKG4/00-00-05.jpg)
![Screenshot at 00:21: A guest, wearing a plaid shirt, begins explaining the relationship between ICL and SF/AGI.](https://ss.rapidrecap.app/screens/ohRQpK_CKG4/00-00-21.jpg)
![Screenshot at 02:03: A comparison table contrasting Continual Learning \(CL\) and In-Context Learning \(ICL\) across five features \(Definition, Weight Updates, Persistence, Compute Cost, Capacity, Data Distribution\).](https://ss.rapidrecap.app/screens/ohRQpK_CKG4/00-02-03.jpg)
![Screenshot at 04:48: The abstract slide for the paper "S-LoRA: Serving Thousands of Concurrent LoRA Adapters" is displayed, detailing scalable serving of LoRA adapters.](https://ss.rapidrecap.app/screens/ohRQpK_CKG4/00-04-48.jpg)