Autoresearch, Agent Loops and the Future of Work

Quick Overview

Andrej Karpathy's "autoresearch" project establishes an agent loop that autonomously performs research by iterating on code, testing results, and committing improvements, suggesting the future of work involves AI agents handling complex, iterative tasks like software development and scientific research.

Key Points: Karpathy's autoresearch project involves an agent loop where the AI agent iterates on code, trains a small language model (LLM) for five minutes, checks the score, and then either commits or discards the result, repeating this process autonomously. The goal of the agent is to engineer agents that make the fastest research progress indefinitely without human involvement, improving metrics like validation loss (e.g., from 0.9979 down to 0.9697 in 126 experiments). The process is codified in a lightweight repository containing about 630 lines of code, where the human only provides the initial context/prompt and the evaluation metric (e.g., Val BPP). The system is designed to prevent human bottlenecks by replacing manual iteration with an autonomous loop that learns from its own results, effectively turning the agent into a self-modifying binary. The concept is generalized beyond LLM training to other domains on the Eval Loop Readiness Map, such as A/B Test Copy Variants and Compiler Optimization, which show high iteration speeds (seconds/minutes) and high evaluation automability. Domains like Political Negotiation and Executive Decision-Making are currently highly subjective and slow (months), representing areas where the full autonomous loop is not yet feasible. The methodology fundamentally shifts human work from writing code/running experiments to programming the agent's research strategy via high-level prompts and guardrails.

Context: The video discusses the implications of Andrej Karpathy's open-sourced "autoresearch" project, which utilizes an autonomous agent loop for iterative research, primarily demonstrated on training a small language model (LLM). Karpathy, formerly Director of AI at Tesla and a founding team member at OpenAI, released this project to showcase how AI can take over the iterative process of scientific discovery, a concept echoed by others like Craig Hewitt and Daniel Messler who see this as the automation of the scientific method.