# Andrej Karpathy: Nanochat Miniseries V1

Source: https://www.youtube.com/watch?v=Jf899oop55Y
Recap page: https://rapidrecap.app/video/Jf899oop55Y
Generated: 2026-01-08T18:04:12.269+00:00

---
## Quick Overview

The core finding from the NanoChat miniseries V1 analysis is that the optimal scaling law ratio (tokens to parameters) for models like GPT-2 and GPT-3 is not a fixed constant, but rather a surprisingly efficient ratio of approximately 8:1 (tokens to parameters) for the D181 model, shifting the fundamental scaling law compared to previous models.

**Key Points:**
- The optimal token-to-parameter ratio for the D181 model was found to be approximately 8:1, a significant shift from previous scaling laws.
- The D181 model, with 1.81 billion parameters and 30 layers, achieved a course score comparable to GPT-3 (175B parameters) using only 734 billion tokens, demonstrating high efficiency.
- The cost to train the D181 model was estimated at $0.076 million, which is significantly cheaper than the estimated $1.8 million for a comparable GPT-3 run.
- The researchers found that the optimal ratio of tokens to parameters (D/N) remains constant across the family of models, simplifying the scaling calculation.
- The experimentation showed that a larger model trained for a shorter time (like D181) outperformed a smaller model trained for a longer time (like GPT-2), suggesting a change in the optimal training strategy.
- The human curation element in data selection (as opposed to purely automatic filtering) was deemed critical for achieving high quality, as evidenced by the superior performance despite smaller parameter counts.

![Screenshot at 04:44: The speaker highlights the comparison between the cost of the new model \($0.076 million\) and the older model \($1.8 million\) to emphasize the massive financial efficiency achieved by the new scaling approach.](https://ss.rapidrecap.app/screens/Jf899oop55Y/00-04-44.jpg)

**Context:** Andrej Karpathy discusses the findings from the NanoChat miniseries V1, which involved training several large language models (LLMs) to understand the underlying physics of scaling laws, particularly focusing on the relationship between model size (parameters) and the amount of training data (tokens). The goal was to determine the most computationally efficient way to train high-performing models, challenging the previously assumed constant scaling ratios.

## Detailed Analysis

The video details the results of training several large language models within the NanoChat miniseries, specifically comparing the efficiency of the new D181 model (1.81B parameters, 30 layers) against older benchmarks like GPT-2 and GPT-3. The key revelation is that the optimal scaling ratio (tokens/parameters) is not fixed but appears to favor a specific ratio, found to be around 8:1 for D181, which allowed it to achieve performance comparable to GPT-3 (175B parameters) with significantly less compute—an estimated $0.076 million versus $1.8 million for GPT-3. Furthermore, the research suggests that the optimal ratio of tokens to parameters (D/N) remains constant across the family of models tested. This finding implies that training a larger model for less time can be more efficient than training a smaller model for longer. The experiment also showed that human curation in data selection, rather than purely automated filtering, played a crucial role in achieving high quality, as models trained on human-curated data performed better even when parameter counts were smaller. The discussion concludes by emphasizing that the fundamental scaling law is shifting away from simply counting parameters to optimizing the ratio based on the available compute budget.

### NanoChat V1 Scaling Laws

- D181 achieves GPT-3 level performance with 8:1 token/parameter ratio
- Optimal D/N ratio is constant across model family
- Larger models trained shorter periods are more efficient

### Model Comparison & Cost

- D181 (1.81B params) scored similarly to GPT-3 (175B params) on the D22 score
- D181 cost $0.076M vs GPT-3 cost $1.8M
- D181 used 734B tokens vs GPT-3's unknown token count

### Human Element vs. Automation

- The 'human curator' approach yields better results than purely automated filtering
- Intuitive choices about data quality and curation matter significantly

### Implications for Future Training

- The focus shifts from maximizing parameter count to optimizing the token/parameter ratio for the compute budget
- This finding allows for smarter allocation of compute resources, potentially leading to better performance for less cost.

![Screenshot at 00:00: Video title card displaying the podcast/discussion visual and the call to action 'BECOME A MEMBER TODAY!'](https://ss.rapidrecap.app/screens/Jf899oop55Y/00-00-00.jpg)
![Screenshot at 02:28: Speaker discusses the issue where simply looking at validation loss can be tricky, leading into the importance of scaling laws.](https://ss.rapidrecap.app/screens/Jf899oop55Y/00-02-28.jpg)
![Screenshot at 04:45: Speaker points out the massive cost difference: $0.076 million for the new model versus an estimated $1.8 million for a comparable GPT-3 run.](https://ss.rapidrecap.app/screens/Jf899oop55Y/00-04-45.jpg)
![Screenshot at 06:38: The speaker notes that the optimal DN ratio for the landmark Chinchilla paper was 20:1, contrasting it with the new findings.](https://ss.rapidrecap.app/screens/Jf899oop55Y/00-06-38.jpg)
![Screenshot at 09:55: The speaker reveals the token count for the GPT-3 run was approximately 3.3 trillion tokens, highlighting the data scale difference.](https://ss.rapidrecap.app/screens/Jf899oop55Y/00-09-55.jpg)