# Why Vision Language Models Ignore What They See [Munawar Hayat] - 758 Source: https://www.youtube.com/watch?v=8gm9pXhlzEc Recap page: https://rapidrecap.app/video/8gm9pXhlzEc Generated: 2025-12-09T20:04:38.768+00:00 --- ## Quick Overview Vision Language Models (VLMs) frequently ignore visual input, relying instead on the language model's parametric memory, which researchers at Qualcomm are tackling through architectural changes like injecting visual information via interleaved cross-attention modules and using auxiliary loss functions guided by segmentation masks to enforce visual attention. **Key Points:** - VLMs often fail physics-based generation tasks, such as unstacking boxes, where physical properties change, indicating they rely on parametric memory rather than actual visual content. - Vision foundation models alone show strong spatial correspondence capabilities, but performance drops below chance when combined with language models, suggesting vision gets ignored when the models are merged. - The paper "Attention Guided Alignment in Efficient Vision Language Models" found that similarity distributions between image and text embeddings in the language model space were highly tangled, and attention scores did not prioritize visual tokens corresponding to the object of interest. - The proposed solution involves interleaving cross-attention modules after every fourth block in the language model to inject visual tokens and using an auxiliary loss to maximize attention scores for visually relevant tokens identified via offline computed segmentation masks. - This method of injecting visual tokens via cross-attention instead of concatenation reduces computational complexity from O((m+n)^2) to O(n^2 + mn), making training faster and enabling reasoning over longer videos. - In retrieval research, Generalized Contrastive Learning (GCL) reformulates the loss to combine image, text, and fused image+text embeddings, allowing generalization to composed multimodal queries without collecting new triplet data. - The MultiHuman test bench addresses identity leakage in multi-person generation by defining attention masks as 'islands' where tokens for one person only attend to tokens corresponding to that specific person, preventing cross-identity leakage. **Context:** The host Sam Cherington interviews Munawar Hayat, a researcher at Qualcomm AI Research, about several Qualcomm papers presented at the recent NeurIPS conference focusing on multimodal AI and visual AI. Hayat details his background, starting with image processing and moving into computer vision before joining Qualcomm in 2023 to focus on multimodal generative AI, including vision language models for understanding, generation, and retrieval. ## Detailed Analysis Munawar Hayat discusses significant limitations in current Vision Language Models (VLMs), primarily that they ignore visual data, defaulting to responses based on the language model's parametric memory, which is evident in failures in physics-based generation and spatial reasoning tasks where vision-only models outperform VLMs. His team analyzed this using the paper "Attention Guided Alignment in Efficient Vision Language Models," finding that attention scores failed to focus on relevant visual tokens when predicting the next token. To fix this, they inject visual information at hierarchical levels using cross-attention modules and apply an auxiliary loss based on segmentation masks to force attention onto salient visual regions, which also improves computational efficiency during training compared to standard concatenation methods. Another area explored is composed multimodal retrieval, tackled by the GCL approach, which modifies the loss formulation to utilize combinations of image, text, and fused embeddings, generalizing retrieval capabilities, even to video data, without collecting extensive new training triplets. Furthermore, Hayat introduced the MultiHuman test bench to address identity preservation in multi-person image generation, proposing an architectural constraint that masks attention to prevent tokens belonging to one generated person from attending to tokens of another, thereby avoiding identity leakage. ### VLM Visual Neglect & Physics Failure - Foundational models fail simple physics tasks like unstacking boxes because physical properties change post-action - Language model takes over, responding from parametric memory, not visual content - Humans use mental simulation for affordances, a capability current models lack. ### Attention Analysis in VLMs - Research showed that in VLMs, the similarity distributions of image and text embeddings in the language model space overlap heavily - Analyzing attention scores revealed that visual tokens corresponding to the object of interest were not highly attended to during next token prediction. ### Proposed Solution for Visual Grounding - Researchers interleave cross-attention modules after every fourth block to inject visual tokens throughout the language model layers - An auxiliary loss maximizes attention scores for relevant visual tokens identified by offline segmentation masks. ### Efficiency Gains from Architectural Change - Injecting tokens via cross-attention instead of concatenation reduces compute complexity from O((m+n)^2) to O(n^2 + mn) - This reduction allows for faster training and enables reasoning over longer videos. ### Generalized Contrastive Learning (GCL) for Retrieval - GCL tackles composed multimodal retrieval (e.g., image query + text context) by reformulating the loss matrix to use combinations of image, text, and fused embeddings - This generalizes across modalities without collecting laborious triplet data, even extending to video retrieval benchmarks like COVR. ### Multi-Person Generation Benchmarking - The MultiHuman test bench addresses identity loss in generating multiple subjects, proposing quantifiable metrics like count accuracy and facial identity preservation via a discriminator model. ### Identity Preservation Solution - The solution defines attention masks as 'islands' to ensure tokens for one person only attend to tokens corresponding to that person, preventing identity leakage during generation - This is an inference-only personalization approach, avoiding compute-prohibitive fine-tuning like DreamBooth.