MiMo-V2-Flash Technical Report

Quick Overview

The Mimo-V2-Flash large language model achieves superior performance compared to its rivals by employing a novel architecture that separates reasoning and knowledge processing, resulting in significantly better efficiency and accuracy, particularly in complex reasoning tasks, while maintaining a small model size of 15 billion active parameters.

Key Points: Mimo-V2-Flash is a new large language model designed for maximum power with minimum hardware, featuring 15 billion active parameters. The model's architecture separates reasoning (using a small, 128-token window) from knowledge storage (309 billion pages of expertise). The reasoning component utilizes a technique called Multi-Teacher On Policy Distillation (MT-OPD), which is highly efficient for inference. The model significantly outperforms peers like DeepSeek V3.2, Kimi K2, and GPQA on tough reasoning tasks, achieving 73.4% on SW-Bench (multilingual) and 71.7% on SWA (non-agent tasks). The key innovation is decoupling knowledge from reasoning, preventing the model from losing its core instruction set when the context window is small. The MTP (Multi-Teacher Post-training) method allows the 15B model to perform comparably to much larger models (like 256K context models) without the massive computational burden.

Context: This technical report dives into the architecture and performance metrics of the Mimo-V2-Flash, a new Large Language Model (LLM) developed by Xiaomi. The core focus is on how this model achieves high performance and efficiency despite having a relatively small parameter count compared to competitors, specifically by introducing architectural innovations that optimize the separation of knowledge recall and reasoning processes.

Detailed Analysis

The presentation details the Mimo-V2-Flash, a new 15 billion parameter LLM designed for high performance with low hardware requirements. The core architectural innovation involves splitting the model's functions into three pillars: specialized expert architecture, Multi-Teacher On Policy Distillation (MT-OPD), and the Multi-Teacher Post-training (MTP) method. The specialized expert architecture handles massive amounts of text efficiently by organizing 309 billion pages of knowledge into 256 sections, allowing the model to manage complex reasoning tasks without constantly accessing the entire knowledge base. The reasoning component uses a small, 128-token context window, which keeps the core instructions stable and prevents the model from forgetting fundamental tasks. The MT-OPD technique ensures that the small reasoning window doesn't degrade performance on complex reasoning, leading to better overall results than larger models. Benchmarks confirmed this approach: Mimo-V2-Flash scored 73.4% on SW-Bench and 71.7% on the SWA benchmark, outperforming larger peers like DeepSeek V3.2 and Kimi K2. The MTP method, which uses a small student model to distill knowledge from large expert models, allows for efficient scaling and deployment, resulting in a model that is both fast and highly accurate, especially for coding and reasoning tasks.