# 11: Generative AI – Text-to-Image Models Source: https://www.youtube.com/watch?v=NQBhhRG-Pe4 Recap page: https://rapidrecap.app/video/NQBhhRG-Pe4 Generated: 2026-01-07T16:01:42.773+00:00 --- ## Quick Overview Generative AI image creation relies on text-conditional diffusion models, which work by iteratively denoising pure noise into a coherent image, a process conceptually reversed from adding noise to known images, and text conditioning is achieved by aligning text embeddings (generated via models like CLIP) with image embeddings within a shared latent space. **Key Points:** - Diffusion models, an exciting area of generative AI, generate images by starting with pure noise and iteratively denoising it, requiring many steps initially (e.g., 1,000 steps, now potentially 50-100 steps). - The core training mechanism reverses the forward process: training a network to predict the noise added to an image, or alternatively, to predict the slightly less noisy version of an image. - A practical improvement involves training the model to predict only the noise added to an input image, then iteratively subtracting the predicted noise to achieve the less noisy version, which yields higher quality results. - Text-to-image conditioning relies on creating shared embeddings for text and images using techniques like OpenAI's CLIP (Contrastive Language-Image Pretraining), which maximizes cosine similarity between matching pairs and minimizes it for non-matching pairs across 400 million image-caption pairs. - The U-Net architecture dominates image-to-image tasks, utilizing a contracting path (like a CNN with pooling) and an expansive path (using up-convolutions) connected by residual connections to provide layers with access to information from earlier layers. - CLIP enables zero-shot image classification by embedding a test image and comparing its embedding against the embeddings generated from several candidate text labels, selecting the label whose text embedding is closest to the image embedding. **Context:** The lecture concludes the class by focusing on diffusion models for text-to-image generation, contrasting them briefly with Large Language Models and showcasing examples like Midjourney and OpenAI's Sora text-to-video model. The discussion centers on the mechanics of generating images from noise and the subsequent step of conditioning that generation process using text prompts, which requires establishing a shared representational space between text and images. ## Detailed Analysis The generation of high-quality images via text prompts fundamentally relies on text-conditional diffusion models, which leverage the principle of reversing the noise addition process. Training involves creating pairs of (slightly noisy image, less noisy image) or, more effectively, (image, added noise) to train a denoising network using stochastic gradient descent. The process starts with pure, randomly sampled noise, and the trained model iteratively refines this noise through many steps until a coherent image matching the desired distribution (e.g., stately college buildings) emerges. To control the output with text, the model must be conditioned; this is achieved by ensuring text embeddings and image embeddings reside in the same latent space. OpenAI's CLIP model accomplishes this by training a Transformer text encoder and a ResNet image encoder on 400 million alt-text image pairs, optimizing a loss function that maximizes the cosine similarity of correct image-text pairs (the diagonal) while minimizing the similarity of incorrect pairs (off-diagonals) to prevent model collapse. For image-to-image tasks where the output shape matches the input, the U-Net architecture is dominant, characterized by its symmetric structure of convolutional downsampling followed by upsampling via transposed convolutions, heavily utilizing residual connections to pass fine-grained information across layers. ### Introduction to Generative Models - Diffusion models presented as an exciting area - Examples shown include ChatGPT generating a photorealistic image of students and Midjourney producing a Monet-style village - OpenAI released Sora, a text-to-video model capable of producing one-minute videos from text prompts. ### The Forward and Reverse Diffusion Process - Generating an image starts with pure noise and iteratively denoises it, which is the reverse of the easy-to-perform forward process of adding noise to a known image (like Killian Court) until it is indistinguishable. ### Training the Denoising Network - Training data consists of (input image, output image) pairs where the output is slightly less noisy than the input, allowing the network to learn the necessary function via SGD - A better approach is training the network to predict just the noise component to subtract it iteratively. ### The U-Net Architecture for Image Transformation - Image-to-image processing relies on the U-Net, featuring a contracting left half (CNN/pooling) and an expansive right half (up-convolutions) - Crucial residual connections link corresponding layers on both halves, giving deeper layers access to earlier feature maps. ### Text Conditioning via CLIP - Steering the image generation requires aligning text and image embeddings in a shared space - CLIP (Contrastive Language-Image Pretraining) uses a Transformer stack for text and ResNet for images, trained on 400 million pairs to ensure related text/image embeddings have high cosine similarity. ### Zero-Shot Classification with CLIP - CLIP enables classifying an image against novel labels without retraining by encoding the image and encoding each potential text label, then selecting the label whose embedding is closest to the image embedding.