Nvidia Nemotron ASR... The Whisper Killer?

Quick Overview

Nvidia introduced Neimotron speech ASR at CES 2026, featuring cache-aware streaming for real-time voice processing that eliminates the redundant re-computation inherent in previous buffered inference models, resulting in up to 3x higher efficiency and zero latency drift under maximum load.

Key Points: Neimotron speech ASR employs cache-aware streaming, processing only new audio deltas and maintaining an internal cache of encoder representations across all layers, achieving 3x higher efficiency than buffered systems. The new architecture uses a Fast Conformer with 24 encoder layers plus an RNN transducer decoder, basing it on the 600 million parameter Parakeet models, though it currently outputs only English. The model demonstrates superior throughput, handling 560 concurrent streams at 320-millisecond latency, significantly outperforming the previous 1.1 billion parameter Parakeet CTC model, which handled 180 streams on the same hardware. Neimotron ASR provides zero latency drift, even at maximum load, addressing the critical issue where previous Parakeet models experienced higher latency as load increased. Users can dynamically control the latency mode at inference time without retraining, allowing selection of operating points sacrificing Word Error Rate for much higher throughput and lower latency. When transcribing a 13-minute audio file on a DGX Spark unit, the inference time was about 27 seconds, making it almost 28-29 times faster than real time. Traditional streaming ASR, like that used in older Parakeet models, utilized buffered inference involving a sliding window that forces the model to reread and recompute attention context for overlapping audio frames.

Context: The video analyzes Nvidia's advancement in Automatic Speech Recognition (ASR) following the release of OpenAI's Whisper in 2022, which set the standard for local speech-to-text transcription. Nvidia previously led with local Parakeet models, which, while fast and accurate, suffered from poor performance scaling in streaming applications due to their buffered inference method requiring constant re-reading of previous audio frames, leading to high GPU VRM utilization and latency drift.