# Evolution of LLMs from 2019 to today | Lex Fridman Podcast Source: https://www.youtube.com/watch?v=h21AK9qM1Go Recap page: https://rapidrecap.app/video/h21AK9qM1Go Generated: 2026-02-04T17:36:33.901+00:00 --- ## Quick Overview The evolution of Large Language Models (LLMs) from GPT-2 (2019) to GPT-OSS 20B (2025) primarily involves architectural shifts toward sparsity via Mixture-of-Experts (MoE) layers and incorporating new mechanisms like RoPE and Grouped Query Attention, which offer substantial computational efficiency gains despite maintaining similar fundamental structures, as highlighted by comparing the dense GPT-2 architecture with the sparse GPT-OSS model. **Key Points:** - GPT-OSS 20B (2025) utilizes a Mixture-of-Experts (MoE) layer, replacing the dense FeedForward layers found in GPT-2 XL (2019), enabling only a fraction of experts (e.g., 2 out of 32) to be active per token. - The GPT-OSS 20B model incorporates Grouped Query Attention (GQA) and Rotary Position Embeddings (RoPE), advancements absent in the GPT-2 architecture. - GPT-OSS 20B boasts a larger context length support of 12K tokens compared to GPT-2's 1K tokens. - The embedding dimension for GPT-OSS 20B is 2,048, a significant increase from GPT-2's 1,000. - The transition from dense to sparse models like GPT-OSS allows for larger model sizes (e.g., 20B parameters) while keeping inference compute costs manageable by activating only a small subset of parameters. - The underlying structure, including the core transformer blocks with attention and feedforward components, remains fundamentally similar between the two models, with GPT-OSS being an evolution rather than a complete reinvention (03:03). - The discussion also touched upon the importance of efficient training methods like FP4 training for achieving faster experimentation cycles (05:45). ![Screenshot at 00:03: A side-by-side architectural comparison slide contrasting the dense GPT-2 XL \(2019\) model with the sparse GPT-OSS 20B \(2025\) model, clearly showing the introduction of the MoE layer and Grouped Query Attention in the newer architecture.](https://ss.rapidrecap.app/screens/h21AK9qM1Go/00-00-03.jpg) **Context:** This video features an interview between Lex Fridman and an interviewee (likely Sebastian Raschka, based on the slide source) discussing the architectural evolution of large language models, specifically comparing OpenAI's GPT-2 XL from 2019 with a hypothetical or emerging sparse model, GPT-OSS 20B, projected for 2025. The core of the discussion centers on how architectural changes, particularly the move towards sparsity using Mixture-of-Experts (MoE), have improved efficiency and scale compared to earlier, fully-connected dense transformer models like GPT-2. ## Detailed Analysis The discussion focuses on the architectural evolution of Large Language Models (LLMs) from GPT-2 (2019) to a hypothetical GPT-OSS 20B (2025). The interviewee explains that the fundamental transformer block structure remains, but key changes enhance efficiency. GPT-2 XL uses fully connected FeedForward networks within its transformer blocks, making it computationally expensive for its size (1.5B parameters). GPT-OSS 20B introduces a Mixture-of-Experts (MoE) layer, replacing the dense FeedForward network. This MoE layer routes each token to only a few experts (e.g., 2 out of 32) for computation, achieving resource savings. Furthermore, GPT-OSS incorporates Grouped Query Attention (GQA) and Rotary Position Embeddings (RoPE). While GPT-2 supported a context length of 1,024 tokens and an embedding dimension of 1,000, GPT-OSS supports 12K tokens and a 2,048 embedding dimension. The interviewee notes that while the architecture is fundamentally similar (still a decoder-only transformer), the shift to sparsity allows for massive models like 20B parameters to be trained and run more efficiently than a dense model of comparable size due to reduced compute per forward pass. The interviewee also mentions that low-precision formats like FP4 are increasingly important for speeding up training, allowing for faster iteration and experimentation. ### Architectural Comparison (GPT-2 vs. GPT-OSS) - GPT-2 XL (2019) features dense FeedForward models and standard Multi-Head Attention - GPT-OSS 20B (2025) introduces MoE layers, Grouped Query Attention, and RoPE. ### Sparsity via MoE - GPT-OSS uses a Router to select only a few experts (e.g., 2 out of 32) per token in the MoE layer, drastically reducing active parameters during inference compared to the fully connected layers in GPT-2. ### Model Scaling & Efficiency - GPT-OSS 20B supports a context length of 12K tokens (vs. 1K for GPT-2) and has a higher embedding dimension (2,048 vs. 1,000), achieving scale without prohibitive computational cost due to sparsity. ### Training Advancements - The discussion highlights the impact of low-precision formats (FP4, FP8) on training speed, allowing for faster experimentation and iteration cycles, as shown in the NVIDIA throughput chart (05:45). ### Continuity in Design - Despite major changes like MoE and GQA, the core transformer block (attention mechanism + feedforward) remains the fundamental recurrent structure derived from the original 'Attention Is All You Need' paper (03:03, 03:33). ![Screenshot at 00:03: Comparison slide showing GPT-2 XL \(2019\) structure versus GPT-OSS 20B \(2025\), highlighting the introduction of MoE and GQA in the newer model.](https://ss.rapidrecap.app/screens/h21AK9qM1Go/00-00-03.jpg) ![Screenshot at 00:13: The abstract and diagram from the original 'Attention Is All You Need' paper, establishing the foundational Transformer architecture that both models derive from.](https://ss.rapidrecap.app/screens/h21AK9qM1Go/00-00-13.jpg) ![Screenshot at 00:53: Detailed view of the GPT-OSS 20B architecture, specifically showing the MoE layer structure with a Router directing tokens to one of 32 experts, and the associated FeedForward \(SwiGLU\) module.](https://ss.rapidrecap.app/screens/h21AK9qM1Go/00-00-53.jpg) ![Screenshot at 05:44: NVIDIA Single GPU Dense Throughput chart illustrating the exponential growth in TFLOPs across various floating-point formats \(FP4, FP8, FP16, FP32\) from 2015 to 2026, emphasizing the benefit of lower precision for compute efficiency.](https://ss.rapidrecap.app/screens/h21AK9qM1Go/00-05-44.jpg) ![Screenshot at 02:22: The interviewee describing how LLMs like GPT-OSS avoid wasting computation by not using all experts/knowledge all the time.](https://ss.rapidrecap.app/screens/h21AK9qM1Go/00-02-22.jpg)