# LLaDA2.0: Scaling Up Diffusion Language Models to 100B

Source: https://www.youtube.com/watch?v=1sD8m3cXQCw
Recap page: https://rapidrecap.app/video/1sD8m3cXQCw
Generated: 2025-12-22T16:26:08.553+00:00

---
## Quick Overview

The LLaDA 2.0 architecture successfully scales diffusion language models to 100 billion parameters by employing a three-phase training scheme involving warm-up, stable decay, and final decay, which allows for efficient training and superior performance over AR models, especially in structured tasks and complex reasoning.

**Key Points:**
- LLaDA 2.0 scales diffusion language models up to 100 billion parameters, significantly larger than previous AR models.
- The training utilizes a three-phase schedule: warm-up (block size 1 to 32), stable decay (block size 32 to 4096), and final decay (block size 4096 back down to 32).
- The model achieves a fluency score of 94.51 on the BFC benchmark, consistently outperforming AR models on this measure.
- The final training phase involves using a document-level attention mask to ensure coherence across documents and using a specific loss function (L-conf) to push the model toward sharper, more certain predictions.
- The technique of using a document-level attention mask and the L-conf loss function helps the model avoid generating semantic nonsense.
- LLaDA 2.0 demonstrates a significant advantage in structured domains, such as math proofs and agentic planning, over unidirectional models due to its inherent bidirectional context understanding.
- The speedup from LLaDA 2.0's parallel decoding is 2.1x faster than the comparable AR model, achieving 357 tokens per second versus 170 TPS for the AR baseline.

![Screenshot at 01:13: The researchers successfully scaled the model up to 100 billion parameters using a three-phase training scheme, which is the main technical achievement discussed.](https://ss.rapidrecap.app/screens/1sD8m3cXQCw/00-01-13.jpg)

**Context:** This AI Papers Podcast episode discusses the LLaDA 2.0 model, a major advancement in diffusion language models (DLMs) developed by researchers from institutions including CMU, Renmin University, Westlake, and Hong Kong University of Science and Technology. The core innovation lies in successfully scaling these models to 100 billion parameters while maintaining efficiency and improving performance, particularly in tasks requiring holistic understanding of context like coding and planning.

## Detailed Analysis

The LLaDA 2.0 paper details a significant leap in diffusion language model scaling, achieving 100 billion parameters. The key to this scale-up is a novel three-phase training process. Phase 1 is a warm-up where the block size gradually increases from 1 to 32 tokens. Phase 2 uses a stable decay, increasing the block size up to 4096. Phase 3, the final decay, shrinks the block size back down to 32. This structured approach avoids the catastrophic forgetting seen when simply increasing block sizes in AR models. The researchers also employed a document-level attention mask during training, ensuring the model considers the entire document context rather than just local sequences, which is crucial for complex tasks like math proofs and planning. Furthermore, they introduced an extra loss function, L-conf, during the final phase to encourage sharper predictions and penalize uncertainty, preventing the model from producing semantic nonsense or relying on vague generalizations. In terms of results, LLaDA 2.0 achieved a 2.1x speedup in inference compared to a standard AR model, generating 357 tokens per second versus 170 TPS. It also scored 94.51 on the BFC benchmark, significantly outperforming the AR baseline, demonstrating superior performance in structured tasks where bidirectional context is vital.

### LLaDA 2.0 Architecture

- Scaling to 100B parameters
- Three-phase training (warm-up, stable decay, final decay)
- Document-level attention mask used for context
- Introduction of L-conf loss for sharper predictions

### Performance Metrics

- 2.1x inference speedup over AR models (357 TPS vs 170 TPS)
- 94.51 on BFC benchmark (outperforming AR baseline)
- Superiority in structured tasks (coding, planning) due to bidirectional context

### Training Methodology

- Block size scaled from 1 up to 4096 and back down
- Post-training fine-tuning used for robustness (SFT)
- Final step involves aligning model output with human preferences using DPO

![Screenshot at 00:00: The video opens with an image promoting membership, overlaid on a background representing audio signals, typical of a podcast visual.](https://ss.rapidrecap.app/screens/1sD8m3cXQCw/00-00-00.jpg)
![Screenshot at 01:16: The speaker discusses scaling up to 100 billion parameters, a key metric for the LLaDA 2.0 model.](https://ss.rapidrecap.app/screens/1sD8m3cXQCw/00-01-16.jpg)
![Screenshot at 02:24: A visual reference to the concept of 'catastrophic forgetting' which the new training scheme aims to avoid.](https://ss.rapidrecap.app/screens/1sD8m3cXQCw/00-02-24.jpg)
![Screenshot at 05:37: The speaker details the final practical detail: block size of 32 was the sweet spot for the decay phase.](https://ss.rapidrecap.app/screens/1sD8m3cXQCw/00-05-37.jpg)
![Screenshot at 08:54: The speaker highlights the 2.1x speedup achieved by LLaDA 2.0 compared to the AR model baseline.](https://ss.rapidrecap.app/screens/1sD8m3cXQCw/00-08-54.jpg)