Simon Willison - 2025: The year in LLMs

Quick Overview

The year 2024 was a monumental one for Large Language Models (LLMs), marked by the explosive growth and competitive shift towards open-weight models, culminating in the success of open-source LLMs like Meta's Llama series, which achieved performance benchmarks previously dominated by proprietary models like GPT-4 and Google's Gemini.

Key Points: 2024 saw a massive shift toward open-weight LLMs, challenging the dominance of proprietary models like GPT-4. Meta's Llama models, particularly Llama 3 (released in April), achieved gold medal performance against top proprietary models in various benchmarks. The success of open models like Llama 3, Claude 3 Opus, and Gemini 1.5 Pro has made the open-weight sector a significant competitive landscape. The ability of modern LLMs to perform complex, multi-step reasoning without external tools or data access is a key breakthrough. The emergence of asynchronous coding agents, like those demonstrated in projects using open models, signals a major increase in developer productivity. A significant security risk highlighted was the potential for prompt injection attacks against web browsers and the lack of robust safety mechanisms in early models.

Context: This podcast episode provides a retrospective analysis of the major developments in Large Language Models (LLMs) throughout 2024, focusing heavily on the impact of open-weight models challenging established leaders like OpenAI and Google. The discussion centers on how the accessibility and capability of these open models are reshaping the industry's competitive balance and technical trajectory.

Detailed Analysis

The year 2024 proved to be a monumental year for LLMs, characterized by an explosive shift towards open-weight models that challenged the previous dominance of closed systems like GPT-4. The speaker notes the sheer volume of shifts occurring, surpassing the progress of the previous three years combined. A key element enabling this shift was the development of core capabilities like reasoning, exemplified by models achieving human-like analytical performance, such as solving complex math equations or logical puzzles without external tools. This was demonstrated by models like Meta's Llama 3, which achieved gold medal performance against proprietary models in the International Mathematical Olympiad. The success of open models, particularly those from Meta, Anthropic (Claude 3), and Google (Gemini 1.5 Pro), has created a highly competitive, multi-polar world. The speaker contrasts this with earlier models that primarily regurgitated training data. A major security concern discussed is the rise of prompt injection attacks, where malicious instructions trick agents (like browser extensions) into leaking private data, a risk exacerbated by the quick deployment of these capable models. The inherent security vulnerability in running models in 'YOLO mode' (You Only Live Once) on local machines without oversight is emphasized. The development of asynchronous coding agents capable of complex, multi-step planning and execution, even when running locally, represents a massive productivity gain. The speaker concludes that the competition is now focused on internal efficiency, with Google's reliance on TPUs as a secret advantage over competitors using standard GPUs, driving down operational costs.