Evolution of LLMs from 2019 to today | Lex Fridman Podcast

Quick Overview

The evolution of Large Language Models (LLMs) from GPT-2 (2019) to GPT-OSS 20B (2025) primarily involves architectural shifts toward sparsity via Mixture-of-Experts (MoE) layers and incorporating new mechanisms like RoPE and Grouped Query Attention, which offer substantial computational efficiency gains despite maintaining similar fundamental structures, as highlighted by comparing the dense GPT-2 architecture with the sparse GPT-OSS model.

Key Points: GPT-OSS 20B (2025) utilizes a Mixture-of-Experts (MoE) layer, replacing the dense FeedForward layers found in GPT-2 XL (2019), enabling only a fraction of experts (e.g., 2 out of 32) to be active per token. The GPT-OSS 20B model incorporates Grouped Query Attention (GQA) and Rotary Position Embeddings (RoPE), advancements absent in the GPT-2 architecture. GPT-OSS 20B boasts a larger context length support of 12K tokens compared to GPT-2's 1K tokens. The embedding dimension for GPT-OSS 20B is 2,048, a significant increase from GPT-2's 1,000. The transition from dense to sparse models like GPT-OSS allows for larger model sizes (e.g., 20B parameters) while keeping inference compute costs manageable by activating only a small subset of parameters. The underlying structure, including the core transformer blocks with attention and feedforward components, remains fundamentally similar between the two models, with GPT-OSS being an evolution rather than a complete reinvention (03:03). The discussion also touched upon the importance of efficient training methods like FP4 training for achieving faster experimentation cycles (05:45).

Context: This video features an interview between Lex Fridman and an interviewee (likely Sebastian Raschka, based on the slide source) discussing the architectural evolution of large language models, specifically comparing OpenAI's GPT-2 XL from 2019 with a hypothetical or emerging sparse model, GPT-OSS 20B, projected for 2025. The core of the discussion centers on how architectural changes, particularly the move towards sparsity using Mixture-of-Experts (MoE), have improved efficiency and scale compared to earlier, fully-connected dense transformer models like GPT-2.