Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models

Quick Overview

The researchers successfully implemented a two-stage framework, combining a perception module (Vision Language Model or VLM) and a reasoning module (LLM), to achieve strong performance in multimodal tasks using significantly smaller models compared to their massive counterparts, demonstrating that downscaling intelligence is feasible through strategic architectural separation.

Key Points: The paper introduces a two-stage framework combining a perception module (VLM) and a reasoning module (LLM) to address multimodal challenges with smaller models. The 8-billion parameter model (Quantization Model) outperformed the 1.7-billion parameter reasoning model on visual grounding tasks, despite the smaller model having a 41x reduction in parameters. When shrinking the model, the visual reasoning ability (perception) dropped by 48% on science questions, while the LLM reasoning module only dropped by 38% on reasoning tasks. The proposed solution involves training the perception module (VLM) and the reasoning module (LLM) separately, allowing the LLM to focus purely on reasoning skills without the overhead of visual processing. The small model (0.6B parameters) trained on the combined task performed worse than the large model (8B parameters) that was trained only on reasoning, indicating the separation of concerns is critical. The final setup uses a very small perception LLM (0.6B parameters) and a larger reasoning LLM (1.7B parameters) to achieve competitive performance while being significantly more efficient.

Context: The video discusses research from Stanford by Mark Endo and Serena Levy focusing on overcoming the computational and performance bottlenecks associated with large multimodal models (LMMs). The core challenge explored is how to maintain high performance on tasks requiring both perception (vision) and reasoning when scaling down model size, leading to the development of a novel two-stage framework to separate these cognitive functions.

Detailed Analysis