AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model

Quick Overview

The AndesVL multimodal large language model achieves efficiency for mobile deployment by aggressively quantizing its base model from 16-bit to 4-bit precision, resulting in a 3x speedup in inference while maintaining performance scores (68.4 on MMLU) close to the uncompressed 16-bit model, thereby bridging the gap between large server-side models and on-device intelligence.

Key Points: AndesVL is an efficient multimodal large language model designed for mobile deployment, achieving high performance despite aggressive quantization. The model was quantized from 16-bit precision down to 4-bit precision, resulting in a 3x speedup in inference time. Despite quantization, the model maintained a strong MMLU score of 68.4, closely matching the 16-bit baseline (70.9). The model employs a novel training method that uses a 'Draft and Verify' approach, where a smaller, faster model drafts responses and a larger model verifies them. AndesVL uses Group Relative Policy Optimization (GRPO) for compression and is designed to operate locally on a mobile device, eliminating reliance on the cloud for core functions. The model's architecture allows it to perform complex reasoning, like handling screenshots and navigating app UIs, directly on the device, surpassing the capabilities of simpler models. The overall strategy is to bridge the gap between massive server models and constrained mobile hardware, enabling powerful AI features locally.

Context: The video discusses the technical report for AndesVL, an efficient multimodal Large Language Model (LLM) developed by the Andes VL team at the OPPO AI Center. The core focus is on overcoming the constraints of mobile hardware—like limited memory, battery life, and processing power—to run sophisticated AI tasks, such as image understanding and complex reasoning, directly on a smartphone rather than relying on cloud processing.

Detailed Analysis

The discussion centers on the AndesVL multimodal large language model, which represents a significant step toward powerful on-device AI. The key innovation is aggressive quantization: the model is compressed from 16-bit precision to 4-bit precision, resulting in a 3x speedup during inference while suffering only a minimal performance drop (from 70.9 to 68.4 on the MMLU benchmark). This efficiency is crucial for mobile deployment, as large models typically strain battery and memory. The model uses a two-stage training method involving a faster 'Draft' model and a larger 'Verify' model, specifically employing Group Relative Policy Optimization (GRPO) for compression. This approach allows the model to efficiently handle complex tasks like interpreting screenshots and navigating user interfaces locally, avoiding latency and privacy concerns associated with cloud processing. The success of AndesVL suggests a shift away from relying solely on massive, centralized models toward practical, powerful local intelligence, effectively closing the gap between server-side capabilities and mobile constraints.