# Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM

Source: https://www.youtube.com/watch?v=dL69AnpaBfQ
Recap page: https://rapidrecap.app/video/dL69AnpaBfQ
Generated: 2025-12-08T20:33:20.469+00:00

---
## Quick Overview

The Xmodel-2.5, a 1.3 billion parameter Large Language Model (LLM) from the paper, achieves superior performance on reasoning benchmarks compared to its predecessor, Xmodel 2.0, primarily by adopting the Muon optimizer and employing a modified training schedule that utilizes a smaller batch size, higher learning rate, and significantly longer context window, all while maintaining a compact size.

**Key Points:**
- Xmodel-2.5 (1.3B parameters) outperforms Xmodel 2.0 on reasoning benchmarks, achieving a 4.58 percentage point advantage on MMLU.
- The key innovation is switching the optimizer from AdamW to Muon, which provides better sharpness and allows for superior reasoning performance.
- The training schedule for Xmodel-2.5 featured a late-phase learning rate drop, stable decay curriculum (S1 and S2), and a context length stretched to over 16,000 tokens.
- The model successfully performed zero-shot hyperparameter transfer, proving the robustness of the structure.
- The new training approach significantly reduced the required training data, using only 1.4 trillion tokens compared to the predecessor's 3.8 trillion tokens for similar reasoning gains.
- Xmodel-2.5 achieves this high performance while remaining small (1.3B parameters) and compute-friendly, enabling deployment on local devices or cars.

![Screenshot at 00:13: The speaker highlights the paper's finding that large LLMs \(like those in the 7B to 13B parameter range\) are impractical for local deployment due to cost, setting the stage for why a smaller, efficient model like Xmodel-2.5 is necessary.](https://ss.rapidrecap.app/screens/dL69AnpaBfQ/00-00-13.png)

**Context:** This video discusses the technical details and performance improvements of a new Large Language Model (LLM) called Xmodel-2.5, which has 1.3 billion parameters. The discussion centers on how this model achieves better reasoning capabilities than its predecessor, Xmodel 2.0, by implementing architectural and training schedule modifications, most notably the adoption of the Muon optimizer over AdamW.

## Detailed Analysis

The video explains the advancements in Xmodel-2.5 (1.3B parameters) which significantly improves upon Xmodel 2.0, particularly in reasoning tasks. The core reason for this improvement is the shift from the AdamW optimizer to the Muon optimizer. Muon provides superior sharpness, which is crucial for robust reasoning, especially when dealing with complex, multi-step reasoning problems or recovering from errors. The researchers noted that AdamW, while stable, often requires a larger training dataset to match the performance of Muon-optimized models. Xmodel-2.5 achieved a score of 52.49% on the MMLU benchmark, beating Xmodel 2.0's 47.91% score, representing a 4.58 percentage point lead. Furthermore, the training methodology was highly efficient: Xmodel-2.5 achieved this result using only 1.4 trillion tokens, compared to the 3.8 trillion tokens used for the predecessor's training, demonstrating massive data efficiency. The context length was also stretched significantly, up to over 16,000 tokens, which is vital for long-context reasoning. The training schedule involved an initial warm-up phase with a high learning rate, followed by stable decay phases (S1 and S2), and a final decay phase where the learning rate dropped significantly, which was crucial for the model's final performance. The paper demonstrated that this architectural switch (AdamW to Muon) allowed the smaller model to maintain performance when transferring settings to larger models, proving the robustness of the approach.

### Performance Gains Over Predecessor

- Xmodel-2.5 (1.3B) scored 52.49% on MMLU, beating Xmodel 2.0's 47.91% (a 4.58 point lead)
- Muon optimizer provided superior sharpness and reasoning ability over AdamW
- The performance gain was achieved with 3.4 times less training data (1.4T tokens vs 3.8T tokens).

### Training Innovations

- Adopted a late-phase learning rate decay schedule
- Context length was extended to over 16,000 tokens
- The training schedule involved a warm-up, stable decay (S1/S2), and final decay phases.

### Hardware Efficiency

- Xmodel-2.5 achieved these results using hardware efficient techniques like 8-bit mixed precision training on Nvidia Hopper architecture
- The small size makes it suitable for local deployment on phones or cars.

### Zero-Shot Transfer Success

- The model successfully transferred its hyperparameter settings from a 20M parameter model to the 1.3B model without retraining, proving the method's generality.

![Screenshot at 00:19: The visual representation of the large context window \(over 16,000 tokens\) being discussed as critical for reasoning.](https://ss.rapidrecap.app/screens/dL69AnpaBfQ/00-00-19.png)
![Screenshot at 01:06: A slide contrasting the performance of Xmodel 2.5 \(Muon\) against Xmodel 2.0 \(AdamW\) on reasoning tasks, showing the superior results of the new model.](https://ss.rapidrecap.app/screens/dL69AnpaBfQ/00-01-06.png)
![Screenshot at 02:24: The speaker explicitly names the key difference: switching from AdamW to the Muon optimizer for better performance.](https://ss.rapidrecap.app/screens/dL69AnpaBfQ/00-02-24.png)
![Screenshot at 04:49: A comparison showing the data efficiency, where Xmodel 2.5 achieved better reasoning with significantly fewer training tokens than its predecessor.](https://ss.rapidrecap.app/screens/dL69AnpaBfQ/00-04-49.png)
![Screenshot at 08:38: A final summary slide showing Xmodel 2.5 achieving a 4.58 percentage point lead in reasoning benchmarks over Xmodel 2.0.](https://ss.rapidrecap.app/screens/dL69AnpaBfQ/00-08-38.png)