# Lec 02. How to Train a Neural Net Source: https://www.youtube.com/watch?v=vidCX_dMCu0 Recap page: https://rapidrecap.app/video/vidCX_dMCu0 Generated: 2026-02-11T15:05:52.624+00:00 --- ## Quick Overview Training a neural network involves using optimization techniques like gradient descent or stochastic gradient descent (SGD) to find parameters that minimize a loss function over the data, leveraging concepts like computational graphs and backpropagation for efficient gradient calculation. **Key Points:** - The training process starts with an input data (x), a ground truth (y), and a model chain of layers with parameters (theta) whose values are adjusted to minimize the loss function. - Gradient descent finds optimal parameters by starting on the loss landscape and moving in the direction of the maximal gradient, using a learning rate (eta) for step size. - Stochastic Gradient Descent (SGD) approximates the full gradient by computing it over a subset of data called a batch, which is faster but introduces noise that acts as an implicit regularizer, potentially bouncing the model out of local minima. - Momentum biases gradient steps in the direction of the previous update, helping to reach minima quicker, but too much momentum causes oscillation; Adam is a popular example of momentum-based optimization. - Backpropagation is an efficiency trick using the chain rule to propagate shared terms (gradients) backward through the computation graph, making gradient calculation practical for very large models. - For a linear layer, the forward pass is $W \cdot x$, the input gradient is the output gradient multiplied by the transposed weight matrix ($g_{out} W^T$), and the weight update is the input multiplied by the output gradient ($x \cdot g_{out}^T$). - Although PyTorch's autograd can calculate derivatives for almost any function, a loss landscape's properties—such as being continuous, differentiable, and smooth—significantly affect optimization difficulty. **Context:** Lecture two of deep learning, presented by Sara Beery, focuses on the mechanics of training a neural network, framing the process as an optimization problem. The discussion reviews fundamental concepts like gradient descent and SGD, introduces the structure of computation graphs, and details how backpropagation efficiently calculates the necessary gradients through multi-layer perceptrons and general Directed Acyclic Graphs (DAGs) to update model parameters. ## Detailed Analysis The lecture outlines neural network training as finding an optimal set of parameters (theta) that minimizes a cost function, which is the sum of losses over all training data. Optimization relies on gradient descent, where steps are taken in the direction of the negative gradient scaled by a learning rate ($\eta$). Due to computational complexity with large datasets, Stochastic Gradient Descent (SGD) uses a batch (a subset of data) to approximate the full gradient, introducing beneficial noise that acts as regularization but also high variance, especially with unbalanced data. Momentum is introduced as a mechanism to accelerate convergence by incorporating past update directions, controlled by a hyperparameter alpha, noting that excessive momentum causes oscillation. The discussion then shifts to the structure necessary for calculating these gradients: the computational graph, typically a Directed Acyclic Graph (DAG) in deep learning, where each node is a differentiable functional transformation. Backpropagation is defined as an efficiency trick that propagates error signals (gradients) backward through this graph using the chain rule, reusing shared computations to make gradient calculation tractable. For a linear layer, the forward pass involves matrix multiplication ($W \cdot x$), and the backward pass similarly involves matrix multiplications with transposed weights to find the input gradient and calculate the weight update ($x \cdot g_{out}^T$). The lecture emphasizes that while PyTorch's autograd can compute derivatives for many non-smooth or discontinuous functions, characteristics like continuity, differentiability, and smoothness in the loss landscape (or activation functions like GeLU vs. ReLU) generally make optimization easier. Techniques like gradient clipping help manage exploding gradients caused by overly steep loss regions. ### Optimization Fundamentals - Review of gradient descent using the loss landscape - SGD approximates total gradient via batches, offering faster computation and implicit regularization - Momentum accelerates steps by biasing movement based on past direction. ### Loss Landscape Properties - Differentiable landscapes are key, though PyTorch handles non-differentiable functions; landscapes with multiple local minima or zero/exploding gradients are hard to optimize. ### Computational Graphs - Deep learning primarily uses DAGs where nodes are differentiable functional transformations - Forward pass computes outputs sequentially; backward pass computes error signals iteratively from output to input. ### Backpropagation Mechanics - Backprop calculates the partial derivative of the cost ($J$) with respect to parameters by propagating shared terms backward through the graph via the chain rule, making gradient computation practical. ### Linear Layer Calculus - Forward pass is $y = Wx$; input gradient is $g_{in} = g_{out}W^T$; weight update is proportional to $x g_{out}^T$ multiplied by the learning rate. ### MLP and ReLU Handling - A multilayer perceptron is a sequence of linear layers and activation functions like ReLU - Backpropagation through ReLU involves creating a gating matrix based on forward pass activations to mask gradients where the input was zero. ### Advanced Techniques - Evolution strategies sample perturbations around current parameters to find locally loss-minimizing directions - Gradient clipping scales back gradients if they exceed a threshold to prevent oscillation from steep regions.