Speech Emotion Recognition Leveraging OpenAI’s Whisper Representations and Attentive Pooling Methods

Quick Overview

The Whisper-based approach for speech emotion recognition, utilizing attentive pooling over Whisper representations, successfully achieves state-of-the-art results, significantly outperforming previous models on the Persian dataset by effectively extracting nuanced acoustic information that standard models often miss, such as subtle prosody changes.

Key Points: The research introduces a novel speech emotion recognition method leveraging representations from OpenAI's Whisper model combined with attentive pooling. This Whisper-based approach achieved a weighted accuracy of 89.19% on the Persian emotion dataset, a significant jump over previous models. The method successfully extracted subtle acoustic features, like pitch and loudness variations, that are crucial for distinguishing nuanced emotions. The core challenge addressed was managing the high-dimensional output of Whisper representations to isolate emotion-relevant features, which was solved by attentive pooling. The approach proved highly effective for low-resource languages like Persian, where labeled data is scarce, by leveraging the general linguistic capabilities of Whisper. The computational load is manageable; the Whisper encoder is frozen, and only the smaller projection layer is trained, making deployment feasible even on consumer hardware. The technique successfully differentiates subtle emotional cues, such as distinguishing between genuine happiness and sarcastic happiness, which standard models struggle with.

Context: This video discusses a research paper from the University of Tehran that aims to improve speech emotion recognition (SER) by applying techniques derived from large language models, specifically OpenAI's Whisper, which is typically used for transcription. The core innovation involves using the internal representations generated by Whisper, rather than just its output, and processing them with an attentive pooling mechanism to better capture the acoustic nuances that define human emotion.