# Lec 16. Generative Models: Conditional Models Source: https://www.youtube.com/watch?v=zaMcHuJwe1w Recap page: https://rapidrecap.app/video/zaMcHuJwe1w Generated: 2026-02-11T15:05:45.827+00:00 --- ## Quick Overview Generative models are fundamentally suited for structured prediction problems, which involve outputting high-dimensional structured data like images or text, because they inherently model multimodal distributions and joint dependencies between dimensions, overcoming the limitations of unstructured methods like least squares regression that average outcomes or classification that fails to capture spatial consistency. **Key Points:** - Structured prediction involves outputting high-dimensional structured objects where dimensions are not statistically independent, a characteristic generative models address effectively. - Unstructured prediction methods like least squares regression fail on multimodal data because they minimize squared error, resulting in an average prediction with zero density under the true distribution, as shown in the colorization example where the average of teal and pink was predicted. - Classification methods solve multimodality by using categorical distributions but fail when predicting high-dimensional objects like an entire image because independent per-pixel classification loses joint dependency, leading to inconsistent colorization across surfaces. - Conditional Generative Adversarial Networks (cGANs) use an adversarial game where a discriminator, conditioned on the input x, penalizes outputs that are unrealistic or do not match the input relationship, with patch discriminators proving effective for capturing local structure. - Conditional Variational Autoencoders (CVAEs) model uncertainty better than GANs regarding mode collapse by ensuring the latent variable z encodes only the information missing from the conditioning variable x, allowing sampling from the full posterior distribution. - Autoregressive models, like those used for text generation, are natively conditional generative models because they model the data as a sequence of conditional predictions, P(x_i | x_1, ..., x_{i-1}), which is how prompting language models functions. - Image-to-text mapping utilizes transformers to tokenize different data types (image patches and text) into a unified format, allowing an image encoder followed by an autoregressive transformer (like GPT) to generate sequential text output (image captioning). **Context:** Phillip Isola introduces the third lecture on generative models, focusing on conditional models and their application to structured prediction problems in image and text domains, contrasting them with older, unstructured prediction techniques. Structured prediction is defined as modeling a high-dimensional object X where dimensions do not factorize into independent marginals, which is contrasted with the historical approach of assuming independence among dimensions. ## Detailed Analysis The lecture establishes that generative models excel at structured prediction—predicting high-dimensional data like images or text—because they can model multimodal distributions and joint dependencies, unlike unstructured methods. Unstructured regression fails on bimodal data (like shirt color being teal or pink) by predicting the average, which has zero density under the true distribution; classification handles multimodality but fails on image prediction by treating pixels independently, losing spatial consistency. Generative models like cGANs and cVAEs solve these structural issues. A cGAN uses a discriminator conditioned on the input X to penalize both unrealistic outputs and outputs that violate the input-output relationship, stabilizing optimization with added regression loss and leveraging patch discriminators to focus on local structure. CVAEs improve upon GANs by using a latent variable Z to encode only the uncertainty not specified by the input X, better capturing the full posterior. Furthermore, autoregressive models, such as large language models (LLMs), are inherently conditional, enabling tasks like sentiment classification simply by conditioning the next-token prediction on the input text (prompting). Finally, image-to-text tasks like captioning use transformers to tokenize image patches and text into a common representation, allowing an image encoder to feed into an autoregressive language model for sequential output generation. ### Structured Prediction Definition - Modeling high-dimensional object X where dimensions lack statistical independence - Unstructured prediction assumes independence, leading to factorization of the joint distribution - Examples include image object detection, speech recognition, and protein folding. ### Flaws of Unstructured Prediction - Least squares regression minimizes error by finding the mean, which has zero density for bimodal data like colorization - Classification handles multimodality via categorical distributions but fails to capture joint dependency across spatially related outputs. ### Conditional GANs (cGANs) - Generator G maps input X to output Y, fooling a discriminator D - D must be conditioned on both X and Y to penalize unrealistic outputs and lack of adherence to the input constraint - Patch discriminators improve structure penalty by assessing small overlapping patches rather than the whole image. ### Conditional VAEs (CVAEs) - CVAEs better capture the full posterior than GANs by having latent variable Z encode only information missing from conditioning variable X - Z learns to model all uncertainty not specified by X, enabling sampling of different valid outcomes. ### Text-to-Text Conditional Models - Autoregressive language models are natively conditional, modeling P(x_i - x_1, ..., x_{i-1}) - Conditioning is achieved by prefixing the input X (prompt) to the text sequence, making LLMs powerful for conditional prediction tasks like sentiment analysis. ### Image-to-Text Models - Transformers unify data types by converting everything into tokens with embeddings - Image captioning uses a Vision Transformer (ViT) encoder to tokenize the image, feeding the resulting tokens into an autoregressive transformer for sequential text generation.