Diffusion Transformers With Representation Autoencoders

Quick Overview

Diffusion Transformers (DiTs) with Representation Autoencoders (RAEs) offer a significant performance leap over standard Variational Autoencoders (VAEs) by achieving a 47x speedup in training and vastly superior semantic fidelity, primarily because the RAE architecture avoids the costly quadratic complexity associated with large discrete latent spaces found in older models.

Key Points: The new Diffusion Transformers with Representation Autoencoders (RAEs) achieve a 47x speedup in training compared to standard VAEs on ImageNet 1K. The RAE approach avoids the bottleneck of previous models by not requiring the latent space width to match the token dimension, which is computationally expensive. The RAE decoder is trained on clean latent codes (analogous to 'null dollars') and then later adapted to handle noisy latent codes from the diffusion process. The RAE architecture produces latent spaces that are both shallow (32x32x4) and wide enough to encode the necessary high-dimensional information for high-fidelity reconstruction. The combination of the RAE and the new DDIT architecture results in a 7x training speedup compared to standard DDITs when tested on latent codes. When tested on ImageNet at 512x512 resolution, the RAE-based model achieved an FID score of 1.13, drastically outperforming the standard DDIT's 4.81 FID score. The authors suggest the RAE approach should become the new default choice for training diffusion transformers over older VAE-based methods.

Context: This video discusses a significant architectural improvement in generative AI models, specifically focusing on replacing the standard Variational Autoencoder (VAE) component in Diffusion Transformers (DiTs) with a new Representation Autoencoder (RAE). The context is overcoming the computational bottlenecks and fidelity limitations of using large, fixed-size discrete latent spaces in previous diffusion models for tasks like image generation.

Detailed Analysis

The presentation introduces Diffusion Transformers (DiTs) utilizing Representation Autoencoders (RAEs) as a superior alternative to models relying on standard VAEs, such as those used in latent diffusion models. The primary issue with VAEs is the computational cost associated with large, discrete latent spaces, which scale quadratically with dimension, leading to bottlenecks and limiting semantic richness. The RAE architecture solves this by decoupling the latent width from the token dimension, allowing for a much shallower latent space (e.g., 32x32x4) that still captures high-dimensional information effectively. The RAE decoder is first trained on clean latent codes (termed 'null dollars') and then fine-tuned using a noise-augmented approach to handle the noisy latent codes generated during the diffusion process. This method results in substantial performance gains: the new RAE-based DiT achieves a 47x training speedup compared to standard VAE-based DiTs when evaluated on ImageNet 1K, and an FID score of 1.13 versus 4.81 for the standard DDIT model. Furthermore, the RAE-based approach improves training convergence speed and maintains high semantic quality, making the RAE architecture the recommended default for future diffusion model development.