# Lec 15. Generative Models: Representation Learning Meets Generative Modeling Source: https://www.youtube.com/watch?v=8zzfcYIELdo Recap page: https://rapidrecap.app/video/8zzfcYIELdo Generated: 2026-02-11T15:05:48.611+00:00 --- ## Quick Overview Variational Autoencoders (VAEs) tightly couple generative modeling and representation learning by employing an autoencoder structure where the encoder maps data to a simple, sampleable prior distribution, typically a Gaussian, allowing the decoder to generate new data, effectively modeling the data distribution as an infinite mixture of Gaussians parameterized by a neural network. **Key Points:** - Generative modeling and representation learning are tightly coupled, and VAEs explicitly model both directions simultaneously: representation learning maps data to abstract representations (encoder f), and generative modeling maps representations to data (decoder g). - A VAE is fundamentally an autoencoder where the encoder maps data not to an arbitrary latent space, but specifically to a simple, sampleable distribution, such as a Gaussian, ensuring that new samples can be generated by sampling from this prior and passing through the decoder. - VAEs model the data distribution as an infinite mixture of Gaussians, parameterized by a neural network (g) that maps points from a simple prior distribution (pz, unit Gaussian) to the means and variances of the mixture components. - The objective function for the generative model (p) involves maximizing the marginal likelihood, which requires calculating an intractable integral over the latent variables z, necessitating approximation techniques like Monte Carlo estimation. - The core VAE 'trick' involves using importance sampling by introducing an encoder (f) that models the posterior distribution p(z|x) as a tractable distribution q(z|x), which is often a Gaussian parameterized by the encoder's output (mean and variance). - The final objective function maximized is the Evidence Lower Bound (ELBO), which acts as a lower bound to the data log likelihood and comprises two terms: a reconstruction/likelihood term and a regularization term penalizing the difference between the inferred distribution q(z|x) and the prior p(z), typically measured by KL divergence. **Context:** Phillip Isola introduces Variational Autoencoders (VAEs) as a model that unifies representation learning (mapping data to structured, abstracted representations) and generative modeling (mapping simple representations back to data). The lecture focuses deeply on VAE mechanics, contrasting them with standard autoencoders where the latent space distribution is uncontrolled, leading to difficulties in sampling new data. ## Detailed Analysis The lecture positions generative modeling as mapping a simple prior distribution (like Gaussian noise z) to a complicated data distribution (x), while representation learning is the inverse mapping. A standard autoencoder's decoder acts like a generator, but sampling new data fails because the distribution over the learned latent encodings is unknown and likely complex. VAEs solve this by forcing the encoder to map data onto a simple, structured distribution, specifically a Gaussian, thereby allowing sampling from the prior p(z) through the decoder g(z) to generate new data points. Mathematically, VAEs are interpreted as fitting an infinite mixture of Gaussians to the data distribution, parameterized finitely by a neural network (g) mapping from the prior z-space. Maximizing the likelihood of data under this model requires integrating over the latent variables z, an intractable integral. The VAE framework approximates this using importance sampling, which necessitates modeling the optimal sampling distribution p(z|x). This modeling step introduces the encoder (f), which takes data x and outputs parameters for a tractable distribution q(z|x) (a Gaussian), effectively performing inference. The overall learning objective becomes maximizing the Evidence Lower Bound (ELBO), which is an unbiased estimator of a lower bound on the log likelihood. The ELBO decomposes into a reconstruction term (how well the decoder fits the data given the sampled latent) and a KL divergence term (how closely the inferred latent distribution q(z|x) matches the prior p(z)), thus learning both the encoder (f) and decoder (g) simultaneously. ### Generative vs. Representation Learning - Representation learning maps raw data (x) to structured representations (z) via encoder (f) - Generative modeling maps representations (z) to data (x) via generator (g) - VAEs model both simultaneously. ### The Autoencoder Limitation - Standard autoencoders allow decoding f(x) back to x, but sampling new data fails because the distribution over the latent space of encodings is not guaranteed to be simple or sampleable. ### VAE as Infinite Gaussian Mixture - VAEs model the data distribution as an infinite mixture of Gaussians - The mixture components (mean and variance) are parameterized by a neural network (g) mapping from a simple prior distribution p(z), typically unit Gaussian. ### The Intractable Likelihood - Maximizing data likelihood requires computing an integral (marginalizing over latent variables z), which is intractable, especially in high dimensions where naive Monte Carlo estimation fails due to required exponential samples. ### Importance Sampling Introduction - Importance sampling approximates the intractable integral by sampling from a more informative distribution q(z) instead of the prior p(z) - The optimal importance distribution q(z) is proportional to p(x - z)p(z), closely related to the true posterior p(z - x). ### The Encoder (Inference Network) - To model the optimal importance distribution p(z - x), VAEs train an encoder (f) that maps x to parameters of a tractable distribution q(z - x) (a Gaussian), defining the inference network. ### The ELBO Objective - The final objective is maximizing the Evidence Lower Bound (ELBO), a lower bound on the log likelihood - ELBO consists of a data likelihood term (reconstruction) and a KL divergence term (regularization forcing q(z - x) towards p(z)).