UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization

Quick Overview

The UniAudio 2.0 model introduces a unified approach to audio processing by factorizing audio tokenization into two distinct streams—a low-frequency reasoning token stream and a high-frequency reconstruction token stream—which allows it to effectively handle tasks like speech, music, and sound effects generation simultaneously, achieving competitive or superior performance over previous models like the standard ASR Transformer and Q2.5 Omni.

Key Points: UniAudio 2.0 proposes a unified audio language model using text-aligned factorized audio tokenization. The model separates audio processing into two streams: low-frequency reasoning tokens (for semantics/structure) and high-frequency reconstruction tokens (for acoustic details). The reasoning tokens are highly compressed (e.g., 5 Hz sampling rate), which avoids latency issues associated with high-frequency processing. The model achieves a 19% lower word error rate on the LibriSpeech benchmark compared to standard ASR models and outperforms Q2.5 Omni on several audio generation tasks. The architecture allows the model to effectively handle speech, music, and sound effects generation within a single framework. The reasoning stream is trained to understand context and structure (like an LLM), while the reconstruction stream focuses on acoustic details, texture, and timbre.

Context: This video discusses the research paper 'UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization,' which aims to create a single AI model capable of processing and generating multiple types of audio (speech, music, sound effects) effectively. The core innovation lies in decoupling the model's understanding of linguistic structure (the 'what') from its ability to render high-fidelity audio (the 'how'), mitigating the trade-offs often seen in previous multimodal audio systems.

Detailed Analysis

The UniAudio 2.0 paper presents a unified audio language model designed to solve the historical trade-off in audio AI where models excelled at either language understanding (like text LLMs) or high-fidelity audio generation, but rarely both simultaneously. The key innovation is factorized audio tokenization, splitting the audio representation into two streams: low-frequency reasoning tokens and high-frequency reconstruction tokens. The reasoning tokens, sampled at a very low rate (like 5 Hz), maintain the semantic structure and context, similar to how an LLM processes text. The reconstruction tokens capture the fine-grained acoustic details, timbre, and texture. This separation prevents the reasoning part from being overwhelmed by acoustic noise, allowing it to focus on linguistic structure and context, which are captured by the text alignment. The paper shows that this approach results in superior performance, significantly beating the standard ASR Transformer and even outperforming models like Q2.5 Omni (which used a similar approach but with a 19% higher word error rate on LibriSpeech). The model effectively handles speech, music, and sound effects, showing that the reasoning tokens capture the high-level plan (like 'walking into a haunted house'), while the reconstruction tokens handle the specific sound event (like a creaking door or eerie music). This architecture allows the model to be much smarter about audio generation than previous models that struggled with the latency or complexity of handling both modalities perfectly.