Lec 02. How to Train a Neural Net

Quick Overview

Training a neural network involves using optimization techniques like gradient descent or stochastic gradient descent (SGD) to find parameters that minimize a loss function over the data, leveraging concepts like computational graphs and backpropagation for efficient gradient calculation.

Key Points: The training process starts with an input data (x), a ground truth (y), and a model chain of layers with parameters (theta) whose values are adjusted to minimize the loss function. Gradient descent finds optimal parameters by starting on the loss landscape and moving in the direction of the maximal gradient, using a learning rate (eta) for step size. Stochastic Gradient Descent (SGD) approximates the full gradient by computing it over a subset of data called a batch, which is faster but introduces noise that acts as an implicit regularizer, potentially bouncing the model out of local minima. Momentum biases gradient steps in the direction of the previous update, helping to reach minima quicker, but too much momentum causes oscillation; Adam is a popular example of momentum-based optimization. Backpropagation is an efficiency trick using the chain rule to propagate shared terms (gradients) backward through the computation graph, making gradient calculation practical for very large models. For a linear layer, the forward pass is $W \cdot x$, the input gradient is the output gradient multiplied by the transposed weight matrix ($g{out} W^T$), and the weight update is the input multiplied by the output gradient ($x \cdot g{out}^T$). Although PyTorch's autograd can calculate derivatives for almost any function, a loss landscape's properties—such as being continuous, differentiable, and smooth—significantly affect optimization difficulty.

Context: Lecture two of deep learning, presented by Sara Beery, focuses on the mechanics of training a neural network, framing the process as an optimization problem. The discussion reviews fundamental concepts like gradient descent and SGD, introduces the structure of computation graphs, and details how backpropagation efficiently calculates the necessary gradients through multi-layer perceptrons and general Directed Acyclic Graphs (DAGs) to update model parameters.