11: Generative AI – Text-to-Image Models

Quick Overview

Generative AI image creation relies on text-conditional diffusion models, which work by iteratively denoising pure noise into a coherent image, a process conceptually reversed from adding noise to known images, and text conditioning is achieved by aligning text embeddings (generated via models like CLIP) with image embeddings within a shared latent space.

Key Points: Diffusion models, an exciting area of generative AI, generate images by starting with pure noise and iteratively denoising it, requiring many steps initially (e.g., 1,000 steps, now potentially 50-100 steps). The core training mechanism reverses the forward process: training a network to predict the noise added to an image, or alternatively, to predict the slightly less noisy version of an image. A practical improvement involves training the model to predict only the noise added to an input image, then iteratively subtracting the predicted noise to achieve the less noisy version, which yields higher quality results. Text-to-image conditioning relies on creating shared embeddings for text and images using techniques like OpenAI's CLIP (Contrastive Language-Image Pretraining), which maximizes cosine similarity between matching pairs and minimizes it for non-matching pairs across 400 million image-caption pairs. The U-Net architecture dominates image-to-image tasks, utilizing a contracting path (like a CNN with pooling) and an expansive path (using up-convolutions) connected by residual connections to provide layers with access to information from earlier layers. CLIP enables zero-shot image classification by embedding a test image and comparing its embedding against the embeddings generated from several candidate text labels, selecting the label whose text embedding is closest to the image embedding.

Context: The lecture concludes the class by focusing on diffusion models for text-to-image generation, contrasting them briefly with Large Language Models and showcasing examples like Midjourney and OpenAI's Sora text-to-video model. The discussion centers on the mechanics of generating images from noise and the subsequent step of conditioning that generation process using text prompts, which requires establishing a shared representational space between text and images.