Lec 04. Architectures: Grids

Quick Overview

The lecture introduces machine learning architectures beyond the multilayer perceptron (MLP) by focusing on building hypotheses directly into model structure, emphasizing that better architectures incorporate stronger inductive biases, like those found in Convolutional Neural Networks (CNNs), which leverage translation equivariance suitable for grid-structured data like images.

Key Points: The multilayer perceptron (MLP) benefits include being a universal approximator, simplicity allowing for elegant theory, and being embarrassingly parallel, but suffers from weak inductive biases, making it very sample inefficient or data hungry. Adding structure via a hypothesis space (stronger inductive bias) allows models to generalize better with less data, potentially finding a solution closer to the true model than relying solely on more data. Convolutional Neural Networks (CNNs) introduce the inductive bias of translation equivariance by using large but overlapping patches and parameter sharing, meaning "A bird should look the same no matter where it is in an image." Convolution is defined as a linear, shift-invariant transformation of grid-structured input data, mathematically represented by applying a learned filter (weights W) across local patches. A convolutional layer is structurally equivalent to a standard linear layer whose weight matrix is a sparse Toeplitz matrix, resulting in fewer parameters, less overfitting, and the ability to generalize to different input sizes. Stacking convolutional layers increases the receptive field, and pooling operations like max pooling or mean pooling provide stability across spatial locations and can introduce new invariances when pooled across channels. Filter parameter calculation requires multiplying the spatial extent (K1 K2) by the number of input channels (Cl), and the number of filters equals the number of output channels (Cl+1).

Context: Sara Beery begins a lecture series on machine learning architectures, moving past the basic multilayer perceptron (MLP) to discuss how architectural design can encode structural hypotheses about the data space. The discussion contrasts the limitations of MLPs, such as their data hunger due to weak inductive biases, with the benefits of specialized architectures like Convolutional Neural Networks (CNNs), which are designed explicitly to handle grid-structured data efficiently.