# Small Batch Size Training for LMs: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful

Source: https://www.youtube.com/watch?v=HmxqP1KjGns
Recap page: https://rapidrecap.app/video/HmxqP1KjGns
Generated: 2026-01-02T18:03:21.415+00:00

---
## Quick Overview

The key takeaway is that using small batch sizes (like 1 or 2) with a standard optimizer like Vanilla SGD, while potentially requiring more steps, is significantly more efficient and robust for training Large Language Models (LLMs) than using gradient accumulation, which introduces unnecessary complexity and performance degradation by averaging gradients over many small steps.

**Key Points:**
- Using small batch sizes (e.g., 1 or 2) with Vanilla SGD is highly effective for training LLMs, contrary to the 'golden rule' requiring large batches for stability.
- Gradient accumulation, which averages gradients from small steps, is deemed wasteful because it introduces complexity without the stability benefits it implies, often leading to poor performance.
- The paper suggests that the stability seen in previous large-batch experiments was an artifact of not tuning the learning rate or decay rates correctly for small batches.
- A key finding is that the required memory overhead for the Adam optimizer state is double that of SGD, making small batch sizes with SGD more memory-efficient.
- The proposed heuristic is to use the smallest batch size that still maximizes FLOP utilization (e.g., 32 or 64) for compute efficiency, but for memory efficiency, small batches (1 or 2) with SGD are superior.
- For the 4-billion parameter Gemma model, fine-tuning with small batch size 1 and a learning rate scaled by the second moment (T2 scaling) delivered robust performance, matching full fine-tuning while saving significant memory.

![Screenshot at 00:29: The speaker explains that the myth of needing large batch sizes for stability is being debunked, noting that small batch sizes can achieve good results.](https://ss.rapidrecap.app/screens/HmxqP1KjGns/00-00-29.jpg)

**Context:** The video discusses recent findings in training Large Language Models (LLMs), specifically challenging the long-held belief that large batch sizes are necessary for stable and effective training, particularly when using simple optimization methods like Stochastic Gradient Descent (SGD). The discussion centers on comparing the performance and efficiency of training with very small batch sizes versus using gradient accumulation to simulate larger batches.

## Detailed Analysis

The video argues against the 'golden rule' in LLM training that mandates large batch sizes for stability, presenting evidence that small batch sizes (as small as 1 or 2) trained with Vanilla SGD can achieve high performance without the memory overhead associated with large batches or gradient accumulation. The authors point out that previous failures when using small batches were often due to not properly tuning the learning rate or decay rates, which are critical for stability. They propose that the stability achieved in large-batch training is largely an artifact of not tuning these parameters correctly, rather than an inherent property of large batches. The comparison between SGD and Adam is highlighted: Adam requires twice the memory overhead due to storing two moments for every parameter, whereas SGD requires only the state for the moving average, making smaller batches more feasible with SGD. The paper suggests that the optimal strategy involves using the smallest batch size that still keeps the GPU saturated with computation, but for memory constraints, small batches are clearly advantageous. This approach was successfully demonstrated on a 4-billion parameter Gemma model, where fine-tuning with batch size 1 and a specific T2 scaling rule for the learning rate yielded performance comparable to full fine-tuning, while dramatically reducing memory usage.

### Batch Size Myth Debunked

- Small batch training with Vanilla SGD proves effective for LLMs
- Gradient accumulation is wasteful because it averages gradients, leading to unnecessary complexity and instability.

### Optimization Comparison

- Adam optimizer requires double the memory overhead compared to SGD due to storing two moments per parameter
- SGD with small batches is more memory efficient for large models.

### Heuristic for Batch Size

- Optimal batch size balances compute saturation (often 32 or 64) with memory constraints
- Smallest effective batch size is key for memory savings.

### Case Study

- Gemma 4B Model: Fine-tuning with batch size 1 using SGD and T2 scaling for the learning rate achieved performance matching full fine-tuning, saving significant memory.

### Key Takeaway

- Simplicity in batch size selection (small batches) combined with correct hyperparameter tuning yields robust, efficient models without complex state tracking.

![Screenshot at 00:05: The speaker introduces the idea that large batch training rules are being rewritten.](https://ss.rapidrecap.app/screens/HmxqP1KjGns/00-00-05.jpg)
![Screenshot at 00:14: The speaker explicitly states the need for large batch sizes for stability is the 'golden rule' being challenged.](https://ss.rapidrecap.app/screens/HmxqP1KjGns/00-00-14.jpg)
![Screenshot at 00:40: The visible waveform illustrates the performance volatility often associated with poor training setups.](https://ss.rapidrecap.app/screens/HmxqP1KjGns/00-00-40.jpg)
![Screenshot at 01:08: The speaker introduces the concept of a 'deep dive' into the mechanics of training.](https://ss.rapidrecap.app/screens/HmxqP1KjGns/00-01-08.jpg)
![Screenshot at 02:56: The speaker notes that small batch sizes trained perfectly stably, even down to batch size 1.](https://ss.rapidrecap.app/screens/HmxqP1KjGns/00-02-56.jpg)