Why Vision Language Models Ignore What They See [Munawar Hayat] - 758

Quick Overview

Vision Language Models (VLMs) frequently ignore visual input, relying instead on the language model's parametric memory, which researchers at Qualcomm are tackling through architectural changes like injecting visual information via interleaved cross-attention modules and using auxiliary loss functions guided by segmentation masks to enforce visual attention.

Key Points: VLMs often fail physics-based generation tasks, such as unstacking boxes, where physical properties change, indicating they rely on parametric memory rather than actual visual content. Vision foundation models alone show strong spatial correspondence capabilities, but performance drops below chance when combined with language models, suggesting vision gets ignored when the models are merged. The paper "Attention Guided Alignment in Efficient Vision Language Models" found that similarity distributions between image and text embeddings in the language model space were highly tangled, and attention scores did not prioritize visual tokens corresponding to the object of interest. The proposed solution involves interleaving cross-attention modules after every fourth block in the language model to inject visual tokens and using an auxiliary loss to maximize attention scores for visually relevant tokens identified via offline computed segmentation masks. This method of injecting visual tokens via cross-attention instead of concatenation reduces computational complexity from O((m+n)^2) to O(n^2 + mn), making training faster and enabling reasoning over longer videos. In retrieval research, Generalized Contrastive Learning (GCL) reformulates the loss to combine image, text, and fused image+text embeddings, allowing generalization to composed multimodal queries without collecting new triplet data. The MultiHuman test bench addresses identity leakage in multi-person generation by defining attention masks as 'islands' where tokens for one person only attend to tokens corresponding to that specific person, preventing cross-identity leakage.