To The “Muon”? A Slightly Technical Breakdown of Kimi K2

Quick Overview

Moonshot AI's Kimi K2 language model, trained with the Muon optimizer, achieves state-of-the-art performance on open-source non-thinking models, excelling in agentic capabilities and various benchmarks like Tau2-Bench and LiveCodeBench. The model's training cost is estimated at $20-30 million, utilizing 15.5 trillion tokens and 32.6 billion activated parameters.

Key Points: Kimi K2, a Mixture-of-Experts (MoE) model, achieves state-of-the-art performance among open-source non-thinking models, with strengths in agentic capabilities. It scores highly on benchmarks like Tau2-Bench (66.1), ACEBench (En) (76.5), SWE-Bench Verified (65.8), SWE-Bench Multilingual (47.3), LiveCodeBench v6 (53.7), AIME 2025 (49.5), GPQA-Diamond (75.1), and OJBench (27.1). The model was trained using the Muon optimizer, which improves training stability and efficiency. The training process involved 15.5 trillion tokens and resulted in a cost estimate of $20-30 million, utilizing 32.6 billion activated parameters. Kimi K2 is open-sourced, with a base model and an instruct-tuned version available. The Muon optimizer itself has shown significant improvements, setting new speed records in tasks like NanoGPT speedrunning, improving training speed by 35% compared to AdamW. The paper introducing Muon was released on February 24, 2025, detailing its scalability and effectiveness.

Context: This video breaks down the technical aspects and performance of Kimi K2, a large language model developed by Moonshot AI. It highlights the model's architecture, its training process using the Muon optimizer, and its benchmark results, comparing it to other leading models. The discussion also touches upon the cost of training such a model and the underlying techniques that enable its performance.

Detailed Analysis

Moonshot AI's Kimi K2 is a 1.04 trillion-parameter Mixture-of-Experts (MoE) transformer model with 32 billion activated parameters, designed to excel in agentic capabilities. Trained using the novel Muon optimizer, Kimi K2 achieves state-of-the-art performance across various benchmarks, including Tau2-Bench, ACEBench, SWE-Bench Verified, SWE-Bench Multilingual, LiveCodeBench v6, AIME 2025, GPQA-Diamond, and OJBench. The Muon optimizer, introduced in a paper released on February 24, 2025, has demonstrated significant improvements in training speed and stability, setting new records in tasks like NanoGPT speedrunning by enhancing training speed by 35% over AdamW. The training process for Kimi K2 utilized 15.5 trillion tokens, with an estimated cost of $20-30 million. The model's architecture, which includes 384 experts and a reduced number of attention heads (64 compared to 128 in DeepSeek-V3), contributes to its efficiency and performance. The research highlights that increased sparsity in MoE models consistently lowers training and validation loss, enhancing overall model performance. Kimi K2 is released as open-source, providing researchers and developers with access to its base and instruct-tuned checkpoints.