# Vision Encoders in Vision-Language Models: A Survey

Source: https://www.youtube.com/watch?v=EvzOxri6tBQ
Recap page: https://rapidrecap.app/video/EvzOxri6tBQ
Generated: 2026-01-02T16:07:29.149+00:00

---
## Quick Overview

The choice of training methodology, not the scale of the vision encoder, drives performance in Vision-Language Models (VLMs), evidenced by the modestly sized 400 million parameter SigLip 2 consistently outperforming the 5.9 billion parameter Intern VT6b on most general VLM benchmarks, reframing VLM development toward strategy over raw size.

**Key Points:**
- Training methodology beats encoder scale, as 400 million parameter SigLip 2 outperforms the 5.9 billion parameter Intern VT6b on almost all general VLM benchmarks.
- The traditional VLM pipeline relies on the Vision Transformer (ViT) to chop images into patches, which the encoder converts into visual tokens, which are then mapped to text token space via a projection module before reaching the LLM.
- Contrastive learning paradigm A, exemplified by SigLip 2, achieved success by using a simpler sigmoid loss instead of CLIP's complex softmax loss and incorporating Localized Contrastive Learning (LCA) for dense feature production.
- Paradigm B, self-supervised learning like Dinov2, excels at pure vision tasks like spatial relationships and segmentation because it trains without text, but requires an extra step to bridge to an LLM.
- Paradigm C, LLM-aligned training, exemplified by Intern 6B, showed a poor cost-benefit ratio, consuming 19 times more compute than smaller models for marginal gains of only one or two points on VQA tasks.
- The connector module's design (MLP projection, Q-Former, cross-attention) has a modest impact; a simple MLP projector often works as well as complex compressors if the encoder training is sound.
- The ultimate lesson confirms three principles: training over scale, prioritizing native resolution (like in Navit) for tasks like OCR, and embracing complementarity (multi-encoder fusion) over universality.

**Context:** This discussion centers on a survey analyzing over 70 vision encoders developed between 2023 and 2025, with the primary goal of understanding what drives performance in Vision-Language Models (VLMs). The context highlights a major historical asymmetry where language models (LLMs) scaled dramatically while vision encoders remained stagnant, often reusing small CLIP variants (300 to 600 million parameters), creating the false assumption that the vision component was a solved problem.

## Detailed Analysis

The analysis confirms that strategy in training methodology fundamentally dictates VLM performance, decisively rejecting the previous assumption that scaling the vision encoder size was paramount. The survey establishes that SigLip 2, at 400 million parameters, surpasses the massive 5.9 billion parameter Intern VT6b across general benchmarks, attributing this to superior training objectives like Localized Contrastive Learning (LCA) which enforces precise feature boundaries. The standard VLM architecture involves image patching, encoding into visual tokens via a ViT-based encoder, alignment via a projection module, and final processing by the LLM; however, the quadratic scaling of tokens based on image resolution creates a significant efficiency bottleneck for the LLM. Three training paradigms define current excellence: contrastive learning (SigLip 2), self-supervised learning (Dinov2 for spatial tasks), and LLM-aligned training (Intern 6B, which is computationally expensive). Furthermore, innovations like Navit's native resolution handling via patch packing and adaptive compression techniques like Pyramid Token Pruning (PTP), which uses query importance to prune tokens, address the token bottleneck better than simply scaling the encoder or using complex connector modules like the Q-Former. The conclusion pushes toward specialization, suggesting that multi-encoder fusion (e.g., combining CLIP, SigLip, and Dinov2) achieves high performance on complex tasks but at a significant computational cost, leading to the final practical takeaway: use SigLip 2 for general tasks, native resolution architectures for documents, and fusion only for heavily spatial tasks, while acknowledging the future threat of encoder-free architectures.

### VLM Architecture Pipeline

- Image chopping into patches
- Encoder generating visual tokens
- Projection module aligning tokens in shared space
- LLM backbone generating final answer

### Training Paradigm A

- Contrastive Learning: CLIP established foundation using softmax loss requiring huge batches (32,000)
- SigLip swapped to sigmoid loss for independent pair processing
- SigLip 2 added LCA for dense, localized features, becoming the current standard

### Training Paradigm B & C

- Specialist vs. Scaled Integration: Dinov2 (SSL) excels at pure vision/spatial tasks but needs bridging
- Intern 6B (LLM-aligned) showed massive compute cost (19x) for marginal performance gains on VQA tasks

### Token Efficiency Bottleneck

- Token count scales quadratically with image resolution, creating an LLM processing bottleneck
- Navit introduced patch-and-pack to avoid padding for native resolution
- PTP uses instruction-guided importance to prune tokens adaptively, achieving 50% reduction with minimal accuracy loss

### Connector Module Impact

- MLP projection is simplest and often sufficient if the encoder is high quality
- Q-Former heavily compresses input to a fixed token count
- Empirical studies show connector design matters far less than encoder training quality

### Key Design Principles & Recommendations

- Training strategy over scale is confirmed
- Prioritize native resolution for documents (Navit style)
- Use SigLip 2 for general purpose VLMs due to the best performance-to-efficiency blend
- Fusion (specialization) is best only for complex spatial tasks due to high cost