Mechanisms of Prompt-Induced Hallucination in Vision–Language Models

Quick Overview

Researchers successfully eliminated prompt-induced hallucination (PIH) in Vision-Language Models (VLMs) by identifying and ablating a small, specific set of attention heads located in the early and middle layers of the language model backbone, achieving up to a nearly 95% reduction in catastrophic errors without degrading general performance.

Key Points: The intervention achieved a nearly 95% reduction in PIH failure rates, dropping them from 42-64% down to 1.4-10% after ablating just a handful of attention heads, in one model as few as three. PIH occurs when the true count of objects is five or more; at this critical threshold, the model's behavior reverses, causing the prompt match rate (agreeing with the wrong number) to jump to 80% or 90%. The vulnerability exploits low base confidence, as certainty in the exact correct count plummets when five or more objects are present, allowing the text prompt to hijack the output. The identified PIH heads were consistently located in the early and middle layers of the language model backbone, with seven of the top ten problem heads found in layer zero for one model, suggesting text bias propagates immediately. Ablating these heads fixed the problem through a dual mechanism: inhibiting prompt-based copying pathways and consistently increasing the model's attention flow back toward the image tokens. The fix generalized across domains, reducing color hallucination errors by 40% to 95% when the same heads identified for number counting were ablated on a color prediction task. The study confirmed the issue is inherited textual bias, as identical top PIH heads were found in two models sharing the same language model backbone (Quinn 2 large language model), despite having different vision encoders.

Context: The discussion centers on Prompt-Induced Hallucination (PIH), a maddening failure where Vision-Language Models (VLMs) ignore visual evidence and lie based on the user's text prompt, exemplified by the 'water lily problem' where a model claiming to see four lilies when only three exist abandons visual evidence entirely when prompted incorrectly. Researchers aimed to move beyond documentation to find the precise neural components causing this failure and surgically remove them without damaging the model's overall functionality.