Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM

Quick Overview

The Xmodel-2.5, a 1.3 billion parameter Large Language Model (LLM) from the paper, achieves superior performance on reasoning benchmarks compared to its predecessor, Xmodel 2.0, primarily by adopting the Muon optimizer and employing a modified training schedule that utilizes a smaller batch size, higher learning rate, and significantly longer context window, all while maintaining a compact size.

Key Points: Xmodel-2.5 (1.3B parameters) outperforms Xmodel 2.0 on reasoning benchmarks, achieving a 4.58 percentage point advantage on MMLU. The key innovation is switching the optimizer from AdamW to Muon, which provides better sharpness and allows for superior reasoning performance. The training schedule for Xmodel-2.5 featured a late-phase learning rate drop, stable decay curriculum (S1 and S2), and a context length stretched to over 16,000 tokens. The model successfully performed zero-shot hyperparameter transfer, proving the robustness of the structure. The new training approach significantly reduced the required training data, using only 1.4 trillion tokens compared to the predecessor's 3.8 trillion tokens for similar reasoning gains. Xmodel-2.5 achieves this high performance while remaining small (1.3B parameters) and compute-friendly, enabling deployment on local devices or cars.

Context: This video discusses the technical details and performance improvements of a new Large Language Model (LLM) called Xmodel-2.5, which has 1.3 billion parameters. The discussion centers on how this model achieves better reasoning capabilities than its predecessor, Xmodel 2.0, by implementing architectural and training schedule modifications, most notably the adoption of the Muon optimizer over AdamW.

Detailed Analysis

The video explains the advancements in Xmodel-2.5 (1.3B parameters) which significantly improves upon Xmodel 2.0, particularly in reasoning tasks. The core reason for this improvement is the shift from the AdamW optimizer to the Muon optimizer. Muon provides superior sharpness, which is crucial for robust reasoning, especially when dealing with complex, multi-step reasoning problems or recovering from errors. The researchers noted that AdamW, while stable, often requires a larger training dataset to match the performance of Muon-optimized models. Xmodel-2.5 achieved a score of 52.49% on the MMLU benchmark, beating Xmodel 2.0's 47.91% score, representing a 4.58 percentage point lead. Furthermore, the training methodology was highly efficient: Xmodel-2.5 achieved this result using only 1.4 trillion tokens, compared to the 3.8 trillion tokens used for the predecessor's training, demonstrating massive data efficiency. The context length was also stretched significantly, up to over 16,000 tokens, which is vital for long-context reasoning. The training schedule involved an initial warm-up phase with a high learning rate, followed by stable decay phases (S1 and S2), and a final decay phase where the learning rate dropped significantly, which was crucial for the model's final performance. The paper demonstrated that this architectural switch (AdamW to Muon) allowed the smaller model to maintain performance when transferring settings to larger models, proving the robustness of the approach.