# Mechanisms of Prompt-Induced Hallucination in Vision–Language Models Source: https://www.youtube.com/watch?v=XvuL4FN8B5c Recap page: https://rapidrecap.app/video/XvuL4FN8B5c Generated: 2026-01-18T13:10:35.367+00:00 --- ## Quick Overview Researchers successfully eliminated prompt-induced hallucination (PIH) in Vision-Language Models (VLMs) by identifying and ablating a small, specific set of attention heads located in the early and middle layers of the language model backbone, achieving up to a nearly 95% reduction in catastrophic errors without degrading general performance. **Key Points:** - The intervention achieved a nearly 95% reduction in PIH failure rates, dropping them from 42-64% down to 1.4-10% after ablating just a handful of attention heads, in one model as few as three. - PIH occurs when the true count of objects is five or more; at this critical threshold, the model's behavior reverses, causing the prompt match rate (agreeing with the wrong number) to jump to 80% or 90%. - The vulnerability exploits low base confidence, as certainty in the exact correct count plummets when five or more objects are present, allowing the text prompt to hijack the output. - The identified PIH heads were consistently located in the early and middle layers of the language model backbone, with seven of the top ten problem heads found in layer zero for one model, suggesting text bias propagates immediately. - Ablating these heads fixed the problem through a dual mechanism: inhibiting prompt-based copying pathways and consistently increasing the model's attention flow back toward the image tokens. - The fix generalized across domains, reducing color hallucination errors by 40% to 95% when the same heads identified for number counting were ablated on a color prediction task. - The study confirmed the issue is inherited textual bias, as identical top PIH heads were found in two models sharing the same language model backbone (Quinn 2 large language model), despite having different vision encoders. **Context:** The discussion centers on Prompt-Induced Hallucination (PIH), a maddening failure where Vision-Language Models (VLMs) ignore visual evidence and lie based on the user's text prompt, exemplified by the 'water lily problem' where a model claiming to see four lilies when only three exist abandons visual evidence entirely when prompted incorrectly. Researchers aimed to move beyond documentation to find the precise neural components causing this failure and surgically remove them without damaging the model's overall functionality. ## Detailed Analysis The core finding of the research is that PIH is a mechanistically localized and model-dependent failure rooted in the early language component of the VLM, not an inherent inability to count. The researchers established a numerical tipping point: models are resilient when the true count is low (2-4 objects), but once the count hits five or more, visual integrity collapses, and the model conforms to the prompt with 80-90% certainty. This switch correlates with a sharp drop in the model's internal confidence regarding the exact visual count, allowing the text prompt to become the deciding factor. Using mean ablation, researchers identified and temporarily disabled specific attention heads responsible for PIH. Ablating just a few of these heads across three models reduced failure rates dramatically, sometimes by over 90%, while restoring true count match rates. The location of these destructive heads was remarkably consistent, residing primarily in the early layers (like layer zero) of the language model backbone, suggesting that misleading text information is processed and propagated before proper language-image integration occurs. Furthermore, the identified heads mediate a broad, domain-agnostic tendency to copy text, as demonstrated by successful ablation on a color prediction task, confirming they facilitate a general copying behavior rather than being number-specific. The fix works by both inhibiting copying pathways and increasing attention toward image tokens, though the internal circuit redesign varies by model: one model primarily stopped format copying, another paradoxically increased format adherence while correcting content, and a third showed the most holistic fix by inhibiting all prompt copying and boosting image attention. ### PIH Mechanism and Threshold - The VLM abandons visual evidence when the prompt misaligns with reality, especially when the true object count exceeds four (the critical threshold) - At low counts (2-4), models resist the prompt, showing skepticism, but above four, prompt match rates jump to 80-90% as visual integrity collapses. ### Diagnostic Findings - PIH exploits low base confidence; when counting five or more objects, certainty in the exact count plummets, spreading probability mass to adjacent numbers, making the text the deciding vote. ### Targeted Intervention via Ablation - Researchers used mean ablation to knock out individual attention heads; disabling the specific PIH heads caused failure rates to drop from 42-64% to under 10% in many cases, restoring visual grounding. ### Localization and Inheritance - The top PIH heads consistently reside in the early and middle layers of the language model backbone (e.g., layer zero), indicating the failure is an inherited textual bias from the LLM component, not the vision encoder. ### Generalization of the Fix - Ablating the PIH heads identified for counting tasks also yielded huge improvements (40-95% reduction) in color prediction hallucinations, proving the heads mediate a broad, domain-agnostic copying tendency. ### Dual Correction Mechanism - Ablation stops the bad habit by inhibiting prompt-based copying (often eliminating digit format copying in outputs) and reinforces the good habit by consistently increasing attention flow back toward the image tokens. ### Model-Specific Circuit Redesign - The internal fix varied; one model stopped copying format, another increased reliance on prompt format while correcting content, and a third (Livo 1 vision) achieved the most holistic fix by inhibiting all prompt copying and strongly increasing image attention.