Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLM

Quick Overview

Multimodal Large Language Models (MLLMs) exhibit a strong preference for visual input over audio input when resolving conflicts between modalities, resulting in a significant drop in audio classification accuracy when audio is removed, although targeted supervision using a technique called Laura can help rebalance this reliance and improve performance on audio-only tasks.

Key Points: MLLMs demonstrate visual dominance, failing to correctly identify audio cues (like a bell sound) when visual information (a video of a quiet engine) is present, leading to an incorrect guess of 'dog' instead of 'bell' when only audio is provided. The paper introduces a technique called Laura (a simple, parameter-efficient fine-tuning strategy) to improve multimodal alignment by teaching the model to correctly answer questions based on audio alone. The Gemini 2.0 Omni 7B model achieved 46.6% accuracy on audio classification with zero audio input when tested against a conflicting visual input, significantly lower than its performance with aligned inputs. The 33-billion parameter model, Gemini 2.5 Pro, scored around 25% on the audio classification task when inputs were aligned, but dropped to 4.04% accuracy when audio input was removed during a conflict test. Laura fine-tuning resulted in the 7B model achieving 81.29% accuracy on the audio-only task when conflicting visual cues were present, showing a massive improvement over the baseline. The study confirms that current MLLM architectures often struggle with true multimodal reasoning, relying too heavily on visual dominance, which is problematic for safety-critical applications like medical diagnosis.

Context: This video discusses the findings of a research paper investigating how Multimodal Large Language Models (MLLMs), such as Gemini and GPT variants, integrate and prioritize information from different modalities like vision, audio, and text. The core issue explored is the observed tendency for these models to favor visual data when presented with conflicting information across modalities, a phenomenon the researchers term 'visual dominance' or 'sensory conflict,' which poses risks in real-world applications requiring reliable audio interpretation.