# How modern LLMs work | Lex Fridman Podcast

Source: https://www.youtube.com/watch?v=44mKM4WxSOA
Recap page: https://rapidrecap.app/video/44mKM4WxSOA
Generated: 2026-02-05T01:34:10.756+00:00

---
## Quick Overview

The discussion between Lex Fridman and Sebastian Raschka focuses on the architectural advancements in large language models (LLMs) since GPT-2, highlighting the shift towards Mixture of Experts (MoE) models like GPT-OSS 20B and Qwen3 Next 80B-A3B, which achieve better performance and efficiency than dense models like GPT-2 XL by activating only a fraction of parameters per token.

**Key Points:**
- Modern LLMs, like GPT-OSS 20B and Qwen3 Next 80B-A3B, predominantly use Mixture of Experts (MoE) architecture, contrasting with the dense architecture of older models like GPT-2 XL (00:09, 09:08).
- Raschka notes that MoE models, such as DeepSeek V3/R1 (671 billion parameters), use a router to selectively activate specific experts (sub-models) per token, leading to efficiency gains (08:48, 09:31).
- DeepSeek V3/R1, for instance, has 671B total parameters but only 57B parameters active per inference step, demonstrating the sparsity benefit of MoE (09:09).
- The shift from dense to sparse models allows for faster training and inference because fewer parameters are computed simultaneously (17:34, 18:03).
- The Qwen3 Next 80B-A3B model features a 'Sparse Gated Attention' mechanism, replacing traditional Multi-Head Attention (MHA) to improve efficiency, similar to DeepSeek's Multi-Head Latent Attention (MLA) which reduces memory usage in the KV cache (11:20).
- In open-weight models, Chinese companies are aggressively releasing new models frequently (e.g., DeepSeek timeline from Dec 2024 to Dec 2025 shows multiple releases), often leading US/European models in release cadence (08:46, 09:03).
- A key trend is utilizing lower-precision floating-point formats (FP16, FP8, FP4) on NVIDIA GPUs to increase throughput without significant performance degradation in deep learning workloads (17:24).

![Screenshot at 09:09: Diagram comparing the architecture of DeepSeek V3/R1 \(sparse MoE\) against a dense architecture, highlighting the MoE layer with a router that selectively activates experts, thus achieving resource savings.](https://ss.rapidrecap.app/screens/44mKM4WxSOA/00-09-09.jpg)

**Context:** This segment of the Lex Fridman Podcast features an interview with Sebastian Raschka, an author and machine learning researcher known for his work on Python-based machine learning, deep learning, and large language models (LLMs). The discussion centers on analyzing the architectural evolution of LLMs, comparing older dense models like GPT-2 XL with newer, more efficient sparse models utilizing Mixture of Experts (MoE) techniques.

## Detailed Analysis

The conversation between Lex Fridman and Sebastian Raschka explores the fundamental architectural differences driving modern Large Language Models (LLMs), primarily focusing on the transition from dense models to sparse Mixture of Experts (MoE) architectures. Raschka points out that models like DeepSeek V3/R1 (671 billion parameters) and GPT-OSS 20B utilize MoE layers, where a router intelligently directs tokens to only a few specialized sub-models (experts) for processing, rather than activating the entire network for every token. This sparsity is a major efficiency gain, as evidenced by DeepSeek V3/R1 only activating 57B parameters per inference step out of 671B total parameters. Architecturally, the shift is also seen in attention mechanisms; for instance, GPT-OSS 20B uses 'Grouped Query Attention' and Qwen3 Next uses 'Sparse Gated Attention,' while DeepSeek employs 'Multi-Head Latent Attention' (MLA) to reduce KV cache memory usage compared to standard MHA. Furthermore, the discussion touches upon the increasing utilization of lower-precision floating-point formats (FP8, FP4) supported by newer NVIDIA hardware to boost compute throughput without severe performance drops. Raschka notes that while the core Transformer block structure remains similar across models like GPT-2 and GPT-OSS 20B, the specific implementations and optimizations, particularly in the feed-forward layers and attention mechanisms, lead to significant speed and efficiency improvements in newer models.

### LLM Architectural Shifts

- Transition from dense models (like GPT-2 XL) to sparse Mixture of Experts (MoE) models (like GPT-OSS 20B, DeepSeek V3/R1)
- MoE uses a router to selectively activate experts, leading to computational efficiency gains (09:31).

### Attention Mechanism Evolution

- Newer models employ specialized attention like Multi-Head Latent Attention (MLA) in DeepSeek or Sparse Gated Attention in Qwen3 Next, replacing standard Multi-Head Attention (MHA) to save memory and improve speed (11:14, 12:22).

### Training and Deployment Differences

- Chinese open-weight models show a faster release cadence than Western counterparts, and the ability to run models locally (not just in the cloud) is a key advantage of open-weight LLMs (08:46, 10:09).

### Hardware Optimization

- NVIDIA's increasing support for lower-precision floating-point formats (FP8, FP4) across Tensor Cores improves power efficiency and compute throughput, which benefits deep learning workloads (17:24).

### Model Comparison

- GPT-OSS 20B's architecture is fundamentally similar to GPT-2 but incorporates MoE and other tweaks, resulting in much faster wall-clock training times (18:24).

![Screenshot at 00:02: Lex Fridman Podcast intro screen showing the host against a stylized Earth horizon view.](https://ss.rapidrecap.app/screens/44mKM4WxSOA/00-00-02.jpg)
![Screenshot at 00:12: A chart titled "Recent AI Models \(Released in 2025+\)" listing various LLMs categorized by developer, illustrating the breadth of the current model landscape \(00:14\).](https://ss.rapidrecap.app/screens/44mKM4WxSOA/00-00-12.jpg)
![Screenshot at 00:49: A timeline graphic titled "The Open Model Timeline" detailing release schedules for models from 2024 and 2025, color-coded by origin \(China, US, Europe\) \(00:51\).](https://ss.rapidrecap.app/screens/44mKM4WxSOA/00-00-49.jpg)
![Screenshot at 05:31: A diagram explaining MoE \(Mixture of Experts\) architecture, showing a router directing input tokens to different sub-models \(experts\) for selective processing \(05:33\).](https://ss.rapidrecap.app/screens/44mKM4WxSOA/00-05-31.jpg)
![Screenshot at 11:41: A comparison diagram detailing architectural differences between GPT-2 XL \(dense\) and GPT-OSS 20B \(sparse MoE\), specifically contrasting standard MHA with Grouped Query Attention \(12:22\).](https://ss.rapidrecap.app/screens/44mKM4WxSOA/00-11-41.jpg)