Lec 20. Scaling Laws

Quick Overview

The lecture on Scaling Laws by Phillip Isola emphasizes that performance in deep learning improves predictably as resources (model size, data, compute) scale, following power laws, but diminishing returns occur if only one factor is scaled while others remain fixed, illustrated by the Kaplan et al. (2020) paper suggesting optimal resource allocation for compute-bound regimes.

Key Points: Deep learning performance improvement follows predictable power law relationships with resources like model size (N), dataset size (D), and compute (C). Scaling laws suggest that when scaling N and D simultaneously, performance improves predictably, but diminishing returns appear if only N or D is scaled. The Kaplan et al. (2020) findings show loss scaling as a power law with respect to compute, data, and model size. The compute required for optimal scaling (minimizing loss for a fixed compute budget) suggests allocating compute such that model size, data size, and steps scale according to specific exponents (e.g., params $\propto C^{0.73}$, data $\propto C^{0.27}$). Rosenfeld et al. (2020) extended these findings to show that extrapolation based on power laws is possible across orders of magnitude for both ImageNet and WikiText-103 tasks. The loss function for model size (N) and data size (D) is modeled as a sum of two power laws: $L(N, D) = \left(\frac{Nc}{N}\right)^{\alphaN} + \left(\frac{Dc}{D}\right)^{\alphaD}$.

Context: Phillip Isola's Lecture 20 on Scaling Laws discusses the empirical power-law relationships discovered in deep learning research, primarily referencing the work by Kaplan et al. (2020) and Rosenfeld et al. (2020). The lecture focuses on how model performance (measured by test loss) changes predictably when scaling core resources like model size (parameters, N), dataset size (tokens, D), and training steps (S). A key theme is the discovery that optimal performance requires scaling these factors in tandem, as isolating the scaling of one factor leads to diminishing returns.