Lec 16. Generative Models: Conditional Models

Quick Overview

Generative models are fundamentally suited for structured prediction problems, which involve outputting high-dimensional structured data like images or text, because they inherently model multimodal distributions and joint dependencies between dimensions, overcoming the limitations of unstructured methods like least squares regression that average outcomes or classification that fails to capture spatial consistency.

Key Points: Structured prediction involves outputting high-dimensional structured objects where dimensions are not statistically independent, a characteristic generative models address effectively. Unstructured prediction methods like least squares regression fail on multimodal data because they minimize squared error, resulting in an average prediction with zero density under the true distribution, as shown in the colorization example where the average of teal and pink was predicted. Classification methods solve multimodality by using categorical distributions but fail when predicting high-dimensional objects like an entire image because independent per-pixel classification loses joint dependency, leading to inconsistent colorization across surfaces. Conditional Generative Adversarial Networks (cGANs) use an adversarial game where a discriminator, conditioned on the input x, penalizes outputs that are unrealistic or do not match the input relationship, with patch discriminators proving effective for capturing local structure. Conditional Variational Autoencoders (CVAEs) model uncertainty better than GANs regarding mode collapse by ensuring the latent variable z encodes only the information missing from the conditioning variable x, allowing sampling from the full posterior distribution. Autoregressive models, like those used for text generation, are natively conditional generative models because they model the data as a sequence of conditional predictions, P(xi | x1, ..., x{i-1}), which is how prompting language models functions. Image-to-text mapping utilizes transformers to tokenize different data types (image patches and text) into a unified format, allowing an image encoder followed by an autoregressive transformer (like GPT) to generate sequential text output (image captioning).