# Lec 04. Architectures: Grids Source: https://www.youtube.com/watch?v=bxVkZ4M-hIE Recap page: https://rapidrecap.app/video/bxVkZ4M-hIE Generated: 2026-02-11T15:05:49.848+00:00 --- ## Quick Overview The lecture introduces machine learning architectures beyond the multilayer perceptron (MLP) by focusing on building hypotheses directly into model structure, emphasizing that better architectures incorporate stronger inductive biases, like those found in Convolutional Neural Networks (CNNs), which leverage translation equivariance suitable for grid-structured data like images. **Key Points:** - The multilayer perceptron (MLP) benefits include being a universal approximator, simplicity allowing for elegant theory, and being embarrassingly parallel, but suffers from weak inductive biases, making it very sample inefficient or data hungry. - Adding structure via a hypothesis space (stronger inductive bias) allows models to generalize better with less data, potentially finding a solution closer to the true model than relying solely on more data. - Convolutional Neural Networks (CNNs) introduce the inductive bias of translation equivariance by using large but overlapping patches and parameter sharing, meaning "A bird should look the same no matter where it is in an image." - Convolution is defined as a linear, shift-invariant transformation of grid-structured input data, mathematically represented by applying a learned filter (weights W) across local patches. - A convolutional layer is structurally equivalent to a standard linear layer whose weight matrix is a sparse Toeplitz matrix, resulting in fewer parameters, less overfitting, and the ability to generalize to different input sizes. - Stacking convolutional layers increases the receptive field, and pooling operations like max pooling or mean pooling provide stability across spatial locations and can introduce new invariances when pooled across channels. - Filter parameter calculation requires multiplying the spatial extent (K1 * K2) by the number of input channels (C_l), and the number of filters equals the number of output channels (C_l+1). **Context:** Sara Beery begins a lecture series on machine learning architectures, moving past the basic multilayer perceptron (MLP) to discuss how architectural design can encode structural hypotheses about the data space. The discussion contrasts the limitations of MLPs, such as their data hunger due to weak inductive biases, with the benefits of specialized architectures like Convolutional Neural Networks (CNNs), which are designed explicitly to handle grid-structured data efficiently. ## Detailed Analysis Architectures are necessary to constrain the hypothesis space, allowing models to generalize better than they could with just more data, by embedding a strong inductive bias. The MLP, while simple and parallelizable, requires vast amounts of data because it assumes little about the structure of the function being learned. CNNs address this by assuming local context is important and incorporating translation equivariance; this is achieved through convolution, where a learned filter (kernel) is applied identically across all local patches of the input grid. This parameter sharing means the model learns specific patterns, like edges, and is invariant to where those patterns appear in the input. Computationally, a convolutional layer is a constrained linear layer represented by a sparse Toeplitz matrix, which reduces parameters and allows the model to handle variable input sizes. Stacking layers increases the receptive field, and pooling (like max pooling) adds stability to the output features. The complexity of convolution involves learning weights across input channels, where the parameters per filter are calculated as spatial extent times input channels, and the number of filters determines the output feature channels. ### MLP Limitations and Architectural Necessity - MLP is a universal approximator but is sample inefficient due to weak inductive biases - Adding a stronger inductive bias shrinks the hypothesis space and improves generalization with less data - The ideal scenario combines more data with a better architecture. ### Convolutional Layers as Structure Encoding - CNNs exploit the inductive bias that "the world is somewhat translation invariant" - Convolution is a linear, shift-invariant transformation where filters highlight specific structures - CNN layers are constrained linear layers represented by sparse Toeplitz matrices, enabling parameter sharing and variable input size handling. ### Convolution Mechanics and Parameterization - A filter's output is a weighted sum of local input pixels matched against the filter pattern - The weight matrix structure is sparse, with identical values along diagonals, contrasting with dense MLP matrices - Parameter count per filter equals K1 * K2 * C_l (spatial extent times input channels). ### Stacking Layers and Receptive Field - Stacking convolutional layers with non-linearities creates a nonlinear convolutional filter whose receptive field expands with depth - Increasing receptive field is achieved by larger kernel sizes or adding more depth. ### Pooling and Downsampling - Pooling (max or mean) adds stability to outputs by summarizing local windows, which helps maintain detection of features like edges even with minor input shifts - Strided operations combine convolution, pooling, and downsampling by increasing the filter's step size, trading computation for spatial reduction.