Small Batch Size Training for LMs: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful

Quick Overview

The key takeaway is that using small batch sizes (like 1 or 2) with a standard optimizer like Vanilla SGD, while potentially requiring more steps, is significantly more efficient and robust for training Large Language Models (LLMs) than using gradient accumulation, which introduces unnecessary complexity and performance degradation by averaging gradients over many small steps.

Key Points: Using small batch sizes (e.g., 1 or 2) with Vanilla SGD is highly effective for training LLMs, contrary to the 'golden rule' requiring large batches for stability. Gradient accumulation, which averages gradients from small steps, is deemed wasteful because it introduces complexity without the stability benefits it implies, often leading to poor performance. The paper suggests that the stability seen in previous large-batch experiments was an artifact of not tuning the learning rate or decay rates correctly for small batches. A key finding is that the required memory overhead for the Adam optimizer state is double that of SGD, making small batch sizes with SGD more memory-efficient. The proposed heuristic is to use the smallest batch size that still maximizes FLOP utilization (e.g., 32 or 64) for compute efficiency, but for memory efficiency, small batches (1 or 2) with SGD are superior. For the 4-billion parameter Gemma model, fine-tuning with small batch size 1 and a learning rate scaled by the second moment (T2 scaling) delivered robust performance, matching full fine-tuning while saving significant memory.

Context: The video discusses recent findings in training Large Language Models (LLMs), specifically challenging the long-held belief that large batch sizes are necessary for stable and effective training, particularly when using simple optimization methods like Stochastic Gradient Descent (SGD). The discussion centers on comparing the performance and efficiency of training with very small batch sizes versus using gradient accumulation to simulate larger batches.

Detailed Analysis