# Diffusion Transformers With Representation Autoencoders

Source: https://www.youtube.com/watch?v=SJTOYgSu1qI
Recap page: https://rapidrecap.app/video/SJTOYgSu1qI
Generated: 2025-11-10T13:32:34.194+00:00

---
## Quick Overview

Diffusion Transformers (DiTs) with Representation Autoencoders (RAEs) offer a significant performance leap over standard Variational Autoencoders (VAEs) by achieving a 47x speedup in training and vastly superior semantic fidelity, primarily because the RAE architecture avoids the costly quadratic complexity associated with large discrete latent spaces found in older models.

**Key Points:**
- The new Diffusion Transformers with Representation Autoencoders (RAEs) achieve a 47x speedup in training compared to standard VAEs on ImageNet 1K.
- The RAE approach avoids the bottleneck of previous models by not requiring the latent space width to match the token dimension, which is computationally expensive.
- The RAE decoder is trained on clean latent codes (analogous to 'null dollars') and then later adapted to handle noisy latent codes from the diffusion process.
- The RAE architecture produces latent spaces that are both shallow (32x32x4) and wide enough to encode the necessary high-dimensional information for high-fidelity reconstruction.
- The combination of the RAE and the new DDIT architecture results in a 7x training speedup compared to standard DDITs when tested on latent codes.
- When tested on ImageNet at 512x512 resolution, the RAE-based model achieved an FID score of 1.13, drastically outperforming the standard DDIT's 4.81 FID score.
- The authors suggest the RAE approach should become the new default choice for training diffusion transformers over older VAE-based methods.

![Screenshot at 00:15: The speaker introduces the core argument: the new approach aims to replace the long-standing Variational Autoencoder \(VAE\) component in generative models, which is the central technical problem addressed by the research.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-00-15.png)

**Context:** This video discusses a significant architectural improvement in generative AI models, specifically focusing on replacing the standard Variational Autoencoder (VAE) component in Diffusion Transformers (DiTs) with a new Representation Autoencoder (RAE). The context is overcoming the computational bottlenecks and fidelity limitations of using large, fixed-size discrete latent spaces in previous diffusion models for tasks like image generation.

## Detailed Analysis

The presentation introduces Diffusion Transformers (DiTs) utilizing Representation Autoencoders (RAEs) as a superior alternative to models relying on standard VAEs, such as those used in latent diffusion models. The primary issue with VAEs is the computational cost associated with large, discrete latent spaces, which scale quadratically with dimension, leading to bottlenecks and limiting semantic richness. The RAE architecture solves this by decoupling the latent width from the token dimension, allowing for a much shallower latent space (e.g., 32x32x4) that still captures high-dimensional information effectively. The RAE decoder is first trained on clean latent codes (termed 'null dollars') and then fine-tuned using a noise-augmented approach to handle the noisy latent codes generated during the diffusion process. This method results in substantial performance gains: the new RAE-based DiT achieves a 47x training speedup compared to standard VAE-based DiTs when evaluated on ImageNet 1K, and an FID score of 1.13 versus 4.81 for the standard DDIT model. Furthermore, the RAE-based approach improves training convergence speed and maintains high semantic quality, making the RAE architecture the recommended default for future diffusion model development.

### Motivation for RAE

- Replacing VAEs for superior efficiency and fidelity
- VAEs suffer from quadratic complexity in large discrete latent spaces, creating bottlenecks
- RAE decouples latent width from token dimension, using a shallower latent space (e.g., 32x32x4).

### RAE Training Strategy

- Two-stage process for robust performance
- Stage 1: Train RAE decoder on clean latent codes ('null dollars')
- Stage 2: Fine-tune decoder using Noise Augmented Training (NAT) with Gaussian noise to handle diffusion latent codes.

### Performance Metrics

- Significant improvements over standard DiTs
- 47x training speedup over VAE-based DiTs using latent codes
- FID score of 1.13 achieved on ImageNet vs. 4.81 for standard DDIT.

### Key Architectural Advantages

- Efficiency and quality
- RAE architecture avoids the need to match token dimension to latent width, improving efficiency
- Noise-aware training ensures the decoder maintains semantic quality when processing noisy latent representations.

![Screenshot at 00:03: Deep dive into Diffusion Transformers, introducing the need for architectural changes.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-00-03.png)
![Screenshot at 00:24: Introduction of the core problem: replacing long-standing VAEs due to computational bottlenecks.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-00-24.png)
![Screenshot at 00:55: Speaker confirms that the limitations of VAEs \(like compression issues\) lead to key problems that need fixing.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-00-55.png)
![Screenshot at 01:19: Visual representation of the latent space reduction achieved \(e.g., from 256x256 down to 32x32\).](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-01-19.png)
![Screenshot at 01:45: Comparison of accuracy: Standard VAEs achieve only 8% accuracy on ImageNet 1K using linear probing, highlighting the need for improvement.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-01-45.png)
![Screenshot at 02:24: Demonstration of the RAE solution: Freezing the powerful encoder and training a decoder.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-02-24.png)
![Screenshot at 03:33: Comparing scores: RAE achieves 84.5% FID vs. 2.2 for standard RAEs, showing significant quality gain.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-03-33.png)
![Screenshot at 04:45: The three main fixes identified: RAE architecture, noise-aware training, and dimension-dependent shift.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-04-45.png)
![Screenshot at 06:36: The benefit of the new schedule: Dropping the FID score significantly \(from 4.81 to 1.13\) on ImageNet.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-06-36.png)
![Screenshot at 08:15: Concluding thought on the advantage: The RAE structure allows the model to be shallow yet wide, avoiding computational expense.](https://ss.rapidrecap.app/screens/SJTOYgSu1qI/00-08-15.png)