# State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490 Source: https://www.youtube.com/watch?v=EV7WhVT270Q Recap page: https://rapidrecap.app/video/EV7WhVT270Q Generated: 2026-01-31T23:02:43.129+00:00 --- ## Quick Overview The state of AI in 2026 features intense competition between US and Chinese entities, where technological ideas flow freely, making budget and hardware the main differentiators rather than proprietary knowledge, while the fundamental autoregressive transformer architecture remains largely unchanged, with gains coming from algorithmic tweaks like Mixture of Experts (MoE) and system optimizations like new precision formats (FP8/FP4). **Key Points:** - The DeepSeek moment in early 2025, where DeepSeek-R1 achieved near state-of-the-art performance with less compute, ignited intense competition in AI research and product development. - Sebastian Raschka asserts that no single company holds proprietary technology access in 2026 due to researcher rotation, stating the differentiating factor will be "budget and hardware constraints." - Nathan Lambert notes the current hype surrounds Anthropic's Claude 3.5 Opus, though he acknowledges Gemini 1.5 had a high initial wow factor, and culturally Anthropic excels by betting hard on code. - Chinese open-weight models, including those from Zhipu AI (GLM) and MiniMax (Kimi Moonshot), are gaining prominence, challenging US business models, as they offer unrestricted open-source licenses appealing to users concerned about restrictions on Llama or Gemma. - The fundamental architecture remains the autoregressive transformer derived from GPT-2, with advancements being architectural tweaks like Mixture of Experts (MoE), Multi-head Latent Attention, and system-level gains from utilizing lower precision like FP8/FP4 for faster training. - Users employ different LLMs based on specific tasks: some prefer Gemini for fast tasks or Google search integration, Claude Opus 3.5 for code and philosophical discussion (often with extended thinking), and others use Grok-3 Heavy for hardcore debugging. - Sebastian emphasizes that building models from scratch, as detailed in his books like "Build a Large Language Model from Scratch," is the best way to learn because code verification confirms correctness, unlike mathematical texts which can contain unverified errors. **Context:** This conversation on the Lex Fridman Podcast features machine learning researchers Sebastian Raschka, author of "Build a Large Language Model from Scratch," and Nathan Lambert, post-training lead at AI2 and author on RLHF, discussing the state of AI in 2026. The discussion centers on the acceleration of the AI competition since the 'DeepSeek moment' in early 2025, contrasting the capabilities and strategies of major US-based labs like OpenAI and Google/Gemini with the rapidly evolving open-weight ecosystem dominated by Chinese companies. ## Detailed Analysis The discussion establishes that the AI landscape in 2026 is characterized by rapid acceleration, though fundamental architectural breakthroughs have stalled, with the autoregressive transformer structure persisting. Key gains come from algorithmic refinements like Mixture of Experts (MoE), which packs more knowledge into the network sparsely, and system-level optimizations such as using lower precision (FP8/FP4) to speed up training by increasing tokens per second per GPU. Sebastian Raschka argues that technology access is democratized, making budget and hardware the true winners' metrics, not proprietary ideas. On the model front, Anthropic's Claude 3.5 Opus is currently enjoying massive hype, especially for coding, while Google's Gemini continues to leverage its massive scale and infrastructure advantage (TPUs) against OpenAI, which excels at landing defining research products like Sora and GPT-4o. The open-weight ecosystem, heavily influenced by Chinese releases like DeepSeek, Zhipu AI, and MiniMax, is thriving due to highly permissive licenses, attracting users who prefer local execution or customization over proprietary APIs. A significant development is the focus on tool use, exemplified by models like Jamba, allowing LLMs to reliably use calculators or web searches to mitigate hallucinations, which experts view as a critical unlock for future capability, even if current user habits favor established closed models for critical tasks. ### Competitive Landscape - DeepSeek-R1 kicked off intense competition in 2025; Sebastian believes budget/hardware, not proprietary ideas, will differentiate winners in 2026 - Nathan points to Anthropic's Claude 3.5 Opus currently leading in hype, especially for code capabilities. ### US vs. China Models - Chinese open-weight models (DeepSeek, Kimi, GLM-4) are gaining ground due to unrestricted licenses, potentially undercutting US API subscription models - US models (ChatGPT, Gemini) currently maintain a performance edge in output quality, leading US users to pay for them. ### Architectural Stagnation & Innovation - The core architecture remains the autoregressive transformer derived from GPT-2, with advancements being iterative tweaks like Group Query Attention, RMSNorm, and the widespread adoption of Mixture of Experts (MoE) for sparse activation. ### System & Training Optimizations - Gains come from system-level improvements like using FP8/FP4 precision to increase tokens/second/GPU during training - Post-training focus is high, with supervised fine-tuning and RLHF being key capability unlocks, exemplified by GPT-3 over GPT-2. ### Model Usage Habits - Users exhibit model switching based on task: Gemini for quick explanations, Claude Opus 3.5 for deep code analysis/design thinking, and Grok-3 Heavy for hard debugging - The 'use it until it breaks' mentality dictates model loyalty, similar to choosing a browser. ### Open Source Focus - Chinese models dominate the open space due to unrestricted licenses, unlike Llama or Gemma, which have user limits - Western efforts like AI2's OLMo and NVIDIA's Nemotron are increasing competition in fully open releases, focusing on providing data and code pipelines for reproducibility.