LEMAS:A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Models

Quick Overview

The LEMS paper introduces a massive, extensible multilingual audio suite containing 150,000 hours of audio data across 10 languages, which serves as a high-quality foundation for generative speech models, enabling them to achieve superior performance in areas like precise word-level timing and handling complex multilingual alignment compared to prior methods.

Key Points: LEMS (Large-scale Extensible Multilingual Audio Suite) contains 150,000 hours of audio across 10 languages, including Chinese, English, Russian, German, Latin, and Vietnamese. The dataset's high quality, featuring word-level timings and confidence scores, sets a new benchmark for the entire field of generative speech modeling. The researchers employed two auxiliary objectives during training: CTC alignment loss and an adversarial loss, which prevents the model from falling into pathological repetition loops. The LEMS model achieves high accuracy in tasks like word-level timing, accent classification, and handling nuanced prosody, even with noisy or complex input. The model successfully manages the challenge of aligning text and audio across vastly different writing systems (e.g., Chinese characters vs. Cyrillic/Latin scripts). The process involves first aligning the target language text, then using an auto-regressive model (like LEMS TTS) for generation, which is further refined by an adaptive regeneration mechanism that balances accuracy and naturalness.

Context: This video discusses the LEMS (Large-scale Extensible Multilingual Audio Suite) project, detailed in a research paper, which aims to overcome the data bottleneck that previously limited the scaling of high-quality generative speech models. The core innovation lies in creating a vast, meticulously curated dataset and a robust training pipeline that ensures high precision in multilingual audio synthesis, addressing issues like accurate word timing and accent leakage.

Detailed Analysis

The LEMS project introduced a massive, extensible multilingual audio suite comprising 150,000 hours of audio across 10 distinct languages, including major ones like English, Chinese, Russian, German, and less common ones like Vietnamese. This dataset serves as a high-quality foundation for training generative speech models. The key contributions highlighted are the ability to achieve highly accurate word-level timing and robust performance across diverse linguistic inputs. The researchers achieved this by employing a two-stage training process: first, they aggregated and normalized data from various sources (like Gigaspeech, MLS, Yotiss) into a unified format, ensuring word-level time stamps and confidence scores were present. Second, they introduced techniques to enforce strict alignment, such as using CTC alignment loss and an adversarial loss layer (acting as a reversed current during training) to prevent the model from getting stuck in repetition loops. This approach allows the model to maintain high fidelity in prosody, rhythm, and accent variation, even when processing audio with background noise or non-native accents, making it significantly more robust than previous methods that relied solely on structural or architectural advances.