VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models

Quick Overview

The VisMem system successfully addresses the key problem of large Vision-Language Models (VLMs) struggling with visual reasoning and forgetting visual context by integrating a dual memory system—Short-Term Memory (STM) and Long-Term Memory (LTM)—which significantly improves performance, showing an 11.8% performance boost on visual reasoning and a 43.8% improvement on visual retrieval compared to standard models.

Key Points: VisMem utilizes a dual memory system (STM for immediate context and LTM for semantic knowledge) to overcome visual reasoning limitations in VLMs. The LTM component significantly improves performance, yielding an 11.8% average performance boost on reasoning tasks compared to baseline models. Visual retrieval tasks showed a massive 43.8% improvement when tested across four different unseen test sets using VisMem. The LTM component is semantically dominant, handling abstract concepts and generalizations, while STM handles immediate, fine-grained visual details. The system avoids the costly process of retraining the entire VLM by dynamically inserting memory tokens only when needed, saving significant computational cost. VisMem demonstrated superior performance by maintaining context across longer sequences, avoiding the catastrophic forgetting that plagues older models when trained sequentially on new data.

Context: This video discusses VisMem, a novel architecture designed to enhance the capabilities of Vision-Language Models (VLMs), particularly in tasks requiring visual reasoning and memory retention over extended contexts. The core problem addressed is the tendency for large models to forget visual details from earlier in a sequence, often leading to errors when complex reasoning is required, especially when compared to models relying only on immediate, short-term context.

Detailed Analysis

The discussion focuses on the VisMem architecture, which tackles the limitations of VLMs in retaining visual context for complex reasoning. The solution involves a dual memory system: Short-Term Memory (STM) and Long-Term Memory (LTM). The STM is designed to be visually dominant, capturing immediate, fine-grained details of the current scene, similar to a temporary visual working space. The LTM, conversely, is semantically dominant, storing generalized, abstract knowledge learned from all training data, acting like a reference library. This system allows the model to dynamically access the right memory component based on the task, avoiding the need to constantly re-process the entire visual context, which doubles the inference time. Experiments show VisMem achieves significant gains: an 11.8% improvement on visual reasoning benchmarks and a striking 43.8% improvement on visual retrieval tasks across four unseen test sets compared to a standard baseline model. Furthermore, the LTM system prevents the catastrophic forgetting common in sequentially trained models by allowing the model to retain old knowledge while integrating new information via specific memory tokens, proving much more efficient than full retraining.