# Lec 20. Scaling Laws Source: https://www.youtube.com/watch?v=7hbf4klU3ks Recap page: https://rapidrecap.app/video/7hbf4klU3ks Generated: 2026-02-11T15:31:48.464+00:00 --- ## Quick Overview The lecture on Scaling Laws by Phillip Isola emphasizes that performance in deep learning improves predictably as resources (model size, data, compute) scale, following power laws, but diminishing returns occur if only one factor is scaled while others remain fixed, illustrated by the Kaplan et al. (2020) paper suggesting optimal resource allocation for compute-bound regimes. **Key Points:** - Deep learning performance improvement follows predictable power law relationships with resources like model size (N), dataset size (D), and compute (C). - Scaling laws suggest that when scaling N and D simultaneously, performance improves predictably, but diminishing returns appear if only N or D is scaled. - The Kaplan et al. (2020) findings show loss scaling as a power law with respect to compute, data, and model size. - The compute required for optimal scaling (minimizing loss for a fixed compute budget) suggests allocating compute such that model size, data size, and steps scale according to specific exponents (e.g., params $\propto C^{0.73}$, data $\propto C^{0.27}$). - Rosenfeld et al. (2020) extended these findings to show that extrapolation based on power laws is possible across orders of magnitude for both ImageNet and WikiText-103 tasks. - The loss function for model size (N) and data size (D) is modeled as a sum of two power laws: $L(N, D) = \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D}$. ![Screenshot at 00:14: The introductory slide for Lecture 20, titled 'Scaling Laws,' featuring two graphs illustrating test loss scaling against compute and estimated $S\_{\\text{min}}$ \(related to training steps\), setting the stage for analyzing how loss changes with resource allocation.](https://ss.rapidrecap.app/screens/7hbf4klU3ks/00-00-14.jpg) **Context:** Phillip Isola's Lecture 20 on Scaling Laws discusses the empirical power-law relationships discovered in deep learning research, primarily referencing the work by Kaplan et al. (2020) and Rosenfeld et al. (2020). The lecture focuses on how model performance (measured by test loss) changes predictably when scaling core resources like model size (parameters, N), dataset size (tokens, D), and training steps (S). A key theme is the discovery that optimal performance requires scaling these factors in tandem, as isolating the scaling of one factor leads to diminishing returns. ## Detailed Analysis Phillip Isola presents Lecture 20 focusing on Scaling Laws, which describe how deep learning model performance improves predictably with increased resources. The lecture draws heavily from Kaplan et al. (2020) and Rosenfeld et al. (2020). The first key finding is that loss $L(X)$ scales as a power law with respect to a resource $X$ (data, parameters, or compute): $L(X) = (1/X)^{\alpha} = X^{-\alpha}$. Doubling any resource $X$ results in the loss decreasing by a factor of $2^{-\alpha}$. Empirical evidence from language modeling shows this power law holds across various models, provided the other factors are held constant (00:15-00:38, 01:33-01:41). The Kaplan et al. work on next-word prediction (06:43) and the associated compute formula ($C = 6 \times \text{parameters} \times \text{batchsize} \times \text{steps}$) are used to define optimal resource allocation (01:10-01:21). The second finding emphasizes simultaneous scaling: performance is maximized when model size (N) and dataset size (D) scale together; fixing one while increasing the other leads to diminishing returns (30:01-30:08, 30:54-31:05). The loss function is formally expressed as dependent on both N and D: $L(N, D) = \left[\left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D}\right]^{\alpha_D}$ (31:11-31:25). This leads to conclusions about optimal scaling ratios for a fixed compute budget $C$: model size $\propto C^{0.73}$, data $\propto \text{batch-size} \cdot \text{steps} \propto C^{0.27}$ (34:24-34:46). Finally, the lecture shows Rosenfeld et al.'s work (37:02) demonstrating that these power laws successfully predict generalization error across scales for vision and language tasks, even when extrapolating far beyond measured data points (37:04-37:42). ### Introduction to Scaling Laws - Scaling laws are power law relationships between test loss $L(X)$ and resources $X$ (data, parameters, compute), where doubling $X$ reduces loss by $2^{-\alpha}$ (00:15-00:38). ### Optimal Resource Allocation (Compute Bound) - For a fixed compute budget $C$, optimal scaling suggests allocating resources such that model parameters $\propto C^{0.73}$, data $\propto \text{batch-size} \cdot \text{steps} \propto C^{0.27}$ (34:24-34:46). ### Simultaneous Dependence - Performance scales best when model size (N) and dataset size (D) scale in tandem, as fixing one variable results in diminishing returns (30:01-30:08). The loss function is $L(N, D) = \left[\left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D}\right]^{\alpha_D}$ (31:11-31:25). ### Extrapolation Validity - Rosenfeld et al. (37:02) show that these power law fits accurately predict performance (loss) on held-out data for both ImageNet and WikiText-103 tasks, validating extrapolation capabilities (37:04-37:42). ![Screenshot at 00:14: The introductory slide for Lecture 20, titled 'Scaling Laws,' featuring two graphs illustrating test loss scaling against compute and estimated $S\_{\\text{min}}$ \(related to training steps\), setting the stage for analyzing how loss changes with resource allocation.](https://ss.rapidrecap.app/screens/7hbf4klU3ks/00-00-14.jpg) ![Screenshot at 06:20: A slide posing the core question: 'If you had a given budget of compute, what model would you train on how much data?' \(C vs. D/N trade-off\).](https://ss.rapidrecap.app/screens/7hbf4klU3ks/00-06-20.jpg) ![Screenshot at 33:04: Slide presenting Finding #2: simultaneous dependence on multiple factors, showing the combined loss function $L\(N, D\)$ dependent on both model size \(N\) and data size \(D\), with separate terms for parameter scaling and data scaling.](https://ss.rapidrecap.app/screens/7hbf4klU3ks/00-33-04.jpg) ![Screenshot at 37:03: Slide illustrating the validity of extrapolation based on Rosenfeld et al. \(2020\) work, showing estimated vs. measured errors/loss for ImageNet and WikiText-103 aligning closely to a y=x line.](https://ss.rapidrecap.app/screens/7hbf4klU3ks/00-37-03.jpg) ![Screenshot at 38:12: Slide detailing Finding #3: dependence on model shape, showing that performance depends strongly on scale but weakly on aspect ratio and attention head dimension, with a U-shaped curve for aspect ratio.](https://ss.rapidrecap.app/screens/7hbf4klU3ks/00-38-12.jpg)