Vision Encoders in Vision-Language Models: A Survey

Quick Overview

The choice of training methodology, not the scale of the vision encoder, drives performance in Vision-Language Models (VLMs), evidenced by the modestly sized 400 million parameter SigLip 2 consistently outperforming the 5.9 billion parameter Intern VT6b on most general VLM benchmarks, reframing VLM development toward strategy over raw size.

Key Points: Training methodology beats encoder scale, as 400 million parameter SigLip 2 outperforms the 5.9 billion parameter Intern VT6b on almost all general VLM benchmarks. The traditional VLM pipeline relies on the Vision Transformer (ViT) to chop images into patches, which the encoder converts into visual tokens, which are then mapped to text token space via a projection module before reaching the LLM. Contrastive learning paradigm A, exemplified by SigLip 2, achieved success by using a simpler sigmoid loss instead of CLIP's complex softmax loss and incorporating Localized Contrastive Learning (LCA) for dense feature production. Paradigm B, self-supervised learning like Dinov2, excels at pure vision tasks like spatial relationships and segmentation because it trains without text, but requires an extra step to bridge to an LLM. Paradigm C, LLM-aligned training, exemplified by Intern 6B, showed a poor cost-benefit ratio, consuming 19 times more compute than smaller models for marginal gains of only one or two points on VQA tasks. The connector module's design (MLP projection, Q-Former, cross-attention) has a modest impact; a simple MLP projector often works as well as complex compressors if the encoder training is sound. The ultimate lesson confirms three principles: training over scale, prioritizing native resolution (like in Navit) for tasks like OCR, and embracing complementarity (multi-encoder fusion) over universality.

Context: This discussion centers on a survey analyzing over 70 vision encoders developed between 2023 and 2025, with the primary goal of understanding what drives performance in Vision-Language Models (VLMs). The context highlights a major historical asymmetry where language models (LLMs) scaled dramatically while vision encoders remained stagnant, often reusing small CLIP variants (300 to 600 million parameters), creating the false assumption that the vision component was a solved problem.