Pocket TTS: A High-Quality TTS With Voice Cloning That Runs on CPU (Revised)

Quick Overview

Pocket TTS, a high-quality text-to-speech model, achieves real-time voice cloning on a standard laptop CPU by employing a discrete token method that skips the heavy continuous latent space calculations common in other models, resulting in superior performance and lower latency compared to models like F5 TTS.

Key Points: Pocket TTS achieves high-quality voice cloning and real-time performance running solely on a CPU. The model avoids the computational bottleneck of large, continuous latent space models by using a discrete token approach. Pocket TTS uses a 6-layer student model trained to mimic the output of a larger 1.6 billion parameter teacher model, specifically trained on the audio's discrete token representation. The paper lists various 'training tricks' including sampling at one step per frame and using a technique called 'Gaussian self-distillation' to stabilize training. Pocket TTS achieved a Word Error Rate (WER) of 1.8, significantly beating F5 TTS's WER of 2.21 in comparable tests. The model's ability to distill knowledge from a larger model allows it to run efficiently on consumer hardware like a standard laptop CPU or even a phone.

Context: The video discusses the paper 'Pocket TTS: A High-Quality TTS With Voice Cloning That Runs on CPU (Revised),' which introduces a novel approach to developing efficient text-to-speech (TTS) models capable of voice cloning without relying on massive computational resources like dedicated GPUs or server farms. The discussion centers on the trade-offs between model size, speed, and quality in generative audio models, comparing the new discrete token approach against larger, slower models like F5 TTS and the model it distills from, a 1.6 billion parameter model.

Detailed Analysis

The speakers introduce a fascinating paper detailing Pocket TTS, a text-to-speech (TTS) model capable of voice cloning that runs efficiently on a standard CPU, even a laptop or phone. This contrasts sharply with many state-of-the-art models that require extensive computational resources. The core innovation lies in moving away from continuous latent spaces, which are computationally heavy, towards a discrete token approach. The researchers trained a smaller 6-layer student model to mimic the output of a much larger 1.6 billion parameter teacher model, which was trained using standard diffusion techniques. This distillation process allows the smaller model to achieve high fidelity without the complexity. The paper details several 'training tricks' used, including step sampling (one step per frame) and Gaussian self-distillation, which helped stabilize training and map the audio waveform directly to discrete tokens. Performance metrics show Pocket TTS achieved a Word Error Rate (WER) of 1.8, outperforming F5 TTS (WER of 2.21) while being significantly faster and requiring less computation. The method avoids the iterative refinement common in diffusion models, essentially creating a shortcut to the final output, which results in low latency suitable for real-time applications.