URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding

Quick Overview

The URaG (Unified Retrieval and Generation) framework significantly improves long document understanding in multimodal LLMs by integrating retrieval directly into the early layers, resulting in substantial efficiency gains, including a 44% to 56% reduction in FLOPs for processing long documents compared to external retrieval methods, while maintaining or exceeding state-of-the-art performance on benchmarks like MMLU.

Key Points: URaG achieves a 44% to 56% reduction in FLOPs for processing long documents (scaling from 20 to 100 pages) compared to external retrieval systems. The model performs retrieval by integrating a small retrieval module directly into the early layers (layers 3-6) of the transformer architecture, rather than relying on separate external tools. This internal retrieval mechanism allows URaG to achieve state-of-the-art performance on major benchmarks like MMLU, validating the approach. The core mechanism involves calculating a relevance score for every page against the query using the internal knowledge from early layers, then zeroing in on the top-scoring pages for final processing. URaG shows superior performance on tasks requiring intensive reading, such as answering questions about charts and images embedded within long documents. The design avoids the common pitfalls of external retrieval, such as information interference and quadratic computational costs associated with processing irrelevant data. The model's elegance lies in its minimal overhead; the retrieval module constitutes only 0.05% of the total 3-billion parameter model.

Context: This video discusses a novel framework called URaG (Unified Retrieval and Generation), designed to enhance the ability of multimodal Large Language Models (LLMs) to efficiently process and understand extremely long documents. The primary challenge addressed is the quadratic computational cost and information overload associated with processing extensive context, which traditional methods often tackle inefficiently using external retrieval systems that can introduce noise and latency.