# Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLM Source: https://www.youtube.com/watch?v=wgEEXbvkdmQ Recap page: https://rapidrecap.app/video/wgEEXbvkdmQ Generated: 2025-12-26T19:03:16.217+00:00 --- ## Quick Overview Multimodal Large Language Models (MLLMs) exhibit a strong preference for visual input over audio input when resolving conflicts between modalities, resulting in a significant drop in audio classification accuracy when audio is removed, although targeted supervision using a technique called Laura can help rebalance this reliance and improve performance on audio-only tasks. **Key Points:** - MLLMs demonstrate visual dominance, failing to correctly identify audio cues (like a bell sound) when visual information (a video of a quiet engine) is present, leading to an incorrect guess of 'dog' instead of 'bell' when only audio is provided. - The paper introduces a technique called Laura (a simple, parameter-efficient fine-tuning strategy) to improve multimodal alignment by teaching the model to correctly answer questions based on audio alone. - The Gemini 2.0 Omni 7B model achieved 46.6% accuracy on audio classification with zero audio input when tested against a conflicting visual input, significantly lower than its performance with aligned inputs. - The 33-billion parameter model, Gemini 2.5 Pro, scored around 25% on the audio classification task when inputs were aligned, but dropped to 4.04% accuracy when audio input was removed during a conflict test. - Laura fine-tuning resulted in the 7B model achieving 81.29% accuracy on the audio-only task when conflicting visual cues were present, showing a massive improvement over the baseline. - The study confirms that current MLLM architectures often struggle with true multimodal reasoning, relying too heavily on visual dominance, which is problematic for safety-critical applications like medical diagnosis. ![Screenshot at 04:43: The speaker highlights the core problem: models must be taught to correctly answer questions based on audio when text is misleading, contrasting the visual evidence \(a quiet engine\) with the actual sound \(a dog barking\), which requires a dedicated training approach.](https://ss.rapidrecap.app/screens/wgEEXbvkdmQ/00-04-43.jpg) **Context:** This video discusses the findings of a research paper investigating how Multimodal Large Language Models (MLLMs), such as Gemini and GPT variants, integrate and prioritize information from different modalities like vision, audio, and text. The core issue explored is the observed tendency for these models to favor visual data when presented with conflicting information across modalities, a phenomenon the researchers term 'visual dominance' or 'sensory conflict,' which poses risks in real-world applications requiring reliable audio interpretation. ## Detailed Analysis The discussion centers on the difficulty MLLMs face when presented with conflicting inputs across different modalities, revealing that these models exhibit a strong bias toward visual information. When presented with a video showing a quiet engine but paired with the sound of a dog barking, models often correctly identify the visual but fail to correctly identify the sound if forced to rely only on audio, demonstrating a lack of robust multimodal grounding. The researchers tested models like Gemini 2.0 Omni 7B and Gemini 2.5 Pro (33B parameters) on this conflict test, finding that the larger model scored 25% accuracy when inputs were aligned but dropped to a mere 4.04% when audio was silenced during the test, confirming a heavy reliance on visual cues. To counteract this, the paper proposed a targeted supervision technique called Laura, which focuses on teaching the model to abstain from using visual cues when only audio is provided, or to correctly prioritize the audio stream when a conflict arises. The 7B model, after Laura fine-tuning, saw its audio-only accuracy jump to 81.29% on the conflicting test, proving that architectural fine-tuning can significantly improve reliance on the audio modality, which is crucial for safety-critical applications where visual information might be absent or misleading. ### Introduction to Multimodal Conflict - The problem of visual dominance in MLLMs is introduced using the dog bark vs. quiet engine example - Models prioritize visual processing, even when audio is present and correct, leading to systematic failures when visual cues are misleading. ### Experimental Results (Gemini Models) - Gemini 2.5 Pro (33B) scored 25% accuracy on aligned audio/visual tasks but dropped to 4.04% accuracy when audio was removed in a conflict test - Gemini 2.0 Omni 7B scored 46.6% on audio classification with zero audio input during conflict. ### The Laura Fine-Tuning Strategy - Researchers implemented Laura, a parameter-efficient fine-tuning method, to specifically train models to rely on audio or abstain from guessing when audio is silent - This targeted supervision forces models to learn robust grounding beyond mere visual shortcuts. ### Laura Performance Gains - The 7B model showed massive improvement, jumping from a low baseline to 81.29% accuracy on the audio-only conflict task after Laura tuning - This confirms that architectural guidance can effectively rebalance modality weighting. ### Conclusion on Multimodal Reasoning - The findings suggest that MLLMs often rely on shallow, correlated information rather than deep, truly integrated multimodal reasoning, necessitating better architectural solutions for reliable performance in real-world scenarios. ![Screenshot at 00:00: The video opens with an invitation to 'Become a Member Today!' over an audio waveform graphic, setting the stage for a podcast discussion.](https://ss.rapidrecap.app/screens/wgEEXbvkdmQ/00-00-00.jpg) ![Screenshot at 00:14: The speakers introduce the topic: analyzing how multimodal large language models \(MLLMs\) handle inputs from different modalities \(vision, audio, text\).](https://ss.rapidrecap.app/screens/wgEEXbvkdmQ/00-00-14.jpg) ![Screenshot at 01:04: An example is shown where the model incorrectly identifies a church bell sound as a dog barking when presented with a silent visual of a quiet engine, illustrating sensory conflict.](https://ss.rapidrecap.app/screens/wgEEXbvkdmQ/00-01-04.jpg) ![Screenshot at 02:33: The concept of 'Multimodal Alignment' \(MMA Bench\) is introduced as the benchmark used to evaluate how well models integrate different data streams.](https://ss.rapidrecap.app/screens/wgEEXbvkdmQ/00-02-33.jpg) ![Screenshot at 04:55: The experiment switches to an audio-focused test, asking the model to identify the sound when the visual content is silent, testing its ability to rely solely on audio.](https://ss.rapidrecap.app/screens/wgEEXbvkdmQ/00-04-55.jpg)