On the Role of Discreteness in Diffusion LLMs

Quick Overview

The main finding is that pure auto-regressive (AR) Large Language Models (LLMs) suffer from a structural penalty when trying to maintain local coherence while generating long sequences, as the model tends to lose track of the required structure, leading to a frequency collapse where subsequent token predictions become statistically boring and risk catastrophic information loss, unlike discrete models which can better manage structural constraints.

Key Points: AR LLMs struggle with long-sequence generation because they lose track of structural requirements, leading to frequency collapse. Discrete models, like those using techniques such as CAR (Context-Adaptive Refinement), avoid this issue by maintaining stronger structural constraints. The paper analyzes the failure of AR models in maintaining global coherence, citing the example of predicting 'a' or 'the' correctly but failing to maintain overall structural integrity. The research contrasts AR models, which often rely on simple token-wise objectives, with discrete models that enforce joint objectives across tokens. The core structural challenge is that AR models often fail to enforce constraints like grammar or local coherence across the entire sequence, leading to degradation. One potential mitigation strategy involves using techniques like Context-Adaptive Refinement (CAR) to enforce constraints like discrete token updates and structured loss. The study suggests that future work must align the corruption process (noise injection) with the model's structural requirements to avoid information loss.

Context: This video discusses a fundamental limitation in autoregressive (AR) Large Language Models (LLMs) when they are tasked with generating long, complex sequences, contrasting their performance with discrete generative models. The discussion centers on the concept of 'structural penalty' and 'frequency collapse' observed in AR models when they prioritize immediate token prediction over maintaining global structural coherence, which is easier for discrete models to manage.