# Autoresearch, Agent Loops and the Future of Work Source: https://www.youtube.com/watch?v=nt9j1k2IhUY Recap page: https://rapidrecap.app/video/nt9j1k2IhUY Generated: 2026-03-10T01:32:47.02+00:00 --- ## Quick Overview Andrej Karpathy's "autoresearch" project establishes an agent loop that autonomously performs research by iterating on code, testing results, and committing improvements, suggesting the future of work involves AI agents handling complex, iterative tasks like software development and scientific research. **Key Points:** - Karpathy's autoresearch project involves an agent loop where the AI agent iterates on code, trains a small language model (LLM) for five minutes, checks the score, and then either commits or discards the result, repeating this process autonomously. - The goal of the agent is to engineer agents that make the fastest research progress indefinitely without human involvement, improving metrics like validation loss (e.g., from 0.9979 down to 0.9697 in 126 experiments). - The process is codified in a lightweight repository containing about 630 lines of code, where the human only provides the initial context/prompt and the evaluation metric (e.g., Val BPP). - The system is designed to prevent human bottlenecks by replacing manual iteration with an autonomous loop that learns from its own results, effectively turning the agent into a self-modifying binary. - The concept is generalized beyond LLM training to other domains on the Eval Loop Readiness Map, such as A/B Test Copy Variants and Compiler Optimization, which show high iteration speeds (seconds/minutes) and high evaluation automability. - Domains like Political Negotiation and Executive Decision-Making are currently highly subjective and slow (months), representing areas where the full autonomous loop is not yet feasible. - The methodology fundamentally shifts human work from writing code/running experiments to programming the agent's research strategy via high-level prompts and guardrails. ![Screenshot at 00:08: The title card visually introduces the concept with a retro-style robot interacting with a data stream, symbolizing automated research.](https://ss.rapidrecap.app/screens/nt9j1k2IhUY/00-00-08.jpg) **Context:** The video discusses the implications of Andrej Karpathy's open-sourced "autoresearch" project, which utilizes an autonomous agent loop for iterative research, primarily demonstrated on training a small language model (LLM). Karpathy, formerly Director of AI at Tesla and a founding team member at OpenAI, released this project to showcase how AI can take over the iterative process of scientific discovery, a concept echoed by others like Craig Hewitt and Daniel Messler who see this as the automation of the scientific method. ## Detailed Analysis The video centers on Andrej Karpathy's autoresearch project, which implements an autonomous agent loop to accelerate research, particularly in LLM training. The core mechanism involves an agent that iteratively proposes code changes, trains a small model for a fixed five-minute runtime, evaluates performance using a metric like validation loss (Val BPP), and then either commits the changes (if better) or reverts/discards them. This loop runs continuously, effectively bypassing human iteration speed. The process is contained in a small, lightweight repository (around 630 lines of code) that uses Git to track progress, where the human's role shifts from writing code to defining the initial prompt, the evaluation metric, and the system's guardrails. The concept is shown to be applicable across many domains on an "Eval Loop Readiness Map" (02:42), demonstrating that tasks with fast, objective evaluations (like Code Generation or Compiler Optimization) are already benefiting, while subjective, slow tasks (like Political Negotiation) remain human-centric for now. Other commentators, like Craig Hewitt and Daniel Messler, reinforced this by framing the agent loop as the automation of the scientific method, where the agent defines success, tests hypotheses, and learns from failures, thus reducing bottlenecks inherent in traditional human-led research cycles. ### Karpathy's Autoresearch Project - Establishes an agent loop where the AI agent runs 5-minute LLM training cycles - Agent commits code if improvement is seen, otherwise discards it - The process is self-contained in ~630 lines of code and uses Git for tracking progress. ### The Agent Loop Mechanism - Human defines success (e.g., Val BPP) and the agent iterates indefinitely - Agents read from and write to a shared brain file (LEARNINGS.md) - The loop corrects for state degradation by resetting to the last known good state if an attempt fails. ### Implications for the Future of Work - Shifts human work from writing code to defining high-level strategy/guardrails - Future work involves defining what 'winning' looks like and letting the AI figure out the steps iteratively. ### The Eval Loop Readiness Map - Plots domains based on Iteration Speed (Seconds to Months) vs. Evaluation Automability (Subjective to Fully Automated) - Domains like Code Generation and Compiler Optimization are in the 'sweet spot' (fast iteration, high automation). ### Domains on the Map - Near-Term domains like A/B Test Copy Variants require clear metrics but less human judgment - Fundamentally Human domains like Political Negotiation require subjective, long-term evaluation and are slow to iterate. ![Screenshot at 00:08: The title card visually introduces the concept with a retro-style robot interacting with a data stream, symbolizing automated research.](https://ss.rapidrecap.app/screens/nt9j1k2IhUY/00-00-08.jpg) ![Screenshot at 02:14: The scatter plot from Karpathy's repository showing validation loss improvement \(Y-axis\) over experiments \(X-axis\), illustrating the self-improving nature of the loop.](https://ss.rapidrecap.app/screens/nt9j1k2IhUY/00-02-14.jpg) ![Screenshot at 04:40: The README.md file detailing the setup steps for running a new experiment, emphasizing the separation between human instructions and agent execution.](https://ss.rapidrecap.app/screens/nt9j1k2IhUY/00-04-40.jpg) ![Screenshot at 13:43: The Eval Loop Readiness Map illustrating where various work domains fall based on iteration speed and evaluation automability.](https://ss.rapidrecap.app/screens/nt9j1k2IhUY/00-13-43.jpg)