How modern LLMs work | Lex Fridman Podcast

Quick Overview

The discussion between Lex Fridman and Sebastian Raschka focuses on the architectural advancements in large language models (LLMs) since GPT-2, highlighting the shift towards Mixture of Experts (MoE) models like GPT-OSS 20B and Qwen3 Next 80B-A3B, which achieve better performance and efficiency than dense models like GPT-2 XL by activating only a fraction of parameters per token.

Key Points: Modern LLMs, like GPT-OSS 20B and Qwen3 Next 80B-A3B, predominantly use Mixture of Experts (MoE) architecture, contrasting with the dense architecture of older models like GPT-2 XL (00:09, 09:08). Raschka notes that MoE models, such as DeepSeek V3/R1 (671 billion parameters), use a router to selectively activate specific experts (sub-models) per token, leading to efficiency gains (08:48, 09:31). DeepSeek V3/R1, for instance, has 671B total parameters but only 57B parameters active per inference step, demonstrating the sparsity benefit of MoE (09:09). The shift from dense to sparse models allows for faster training and inference because fewer parameters are computed simultaneously (17:34, 18:03). The Qwen3 Next 80B-A3B model features a 'Sparse Gated Attention' mechanism, replacing traditional Multi-Head Attention (MHA) to improve efficiency, similar to DeepSeek's Multi-Head Latent Attention (MLA) which reduces memory usage in the KV cache (11:20). In open-weight models, Chinese companies are aggressively releasing new models frequently (e.g., DeepSeek timeline from Dec 2024 to Dec 2025 shows multiple releases), often leading US/European models in release cadence (08:46, 09:03). A key trend is utilizing lower-precision floating-point formats (FP16, FP8, FP4) on NVIDIA GPUs to increase throughput without significant performance degradation in deep learning workloads (17:24).

Context: This segment of the Lex Fridman Podcast features an interview with Sebastian Raschka, an author and machine learning researcher known for his work on Python-based machine learning, deep learning, and large language models (LLMs). The discussion centers on analyzing the architectural evolution of LLMs, comparing older dense models like GPT-2 XL with newer, more efficient sparse models utilizing Mixture of Experts (MoE) techniques.