Xmodel-2-history

Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM

<h5 align="center">

</h5>

Model Description

Xmodel-2.5 is a 1.3 billion parameter small language model specifically designed as a lightweight agent core for complex reasoning tasks. The model builds upon Xmodel-2 with four key upgrades:

1. Full μP Support: Extended Megatron-LM to support maximal update parameterization for reliable hyperparameter transfer
2. Efficient Tokenizer: Adopted 129K token DeepSeek-v3 tokenizer for improved compression rate and decoding speed
3. FP8 Mixed Precision: Used E4M3 forward and E5M2 backward FP8 formats to balance precision and throughput
4. Optimizer Scheduling: Switched from AdamW to Muon during decay phase, significantly improving downstream task performance

Trained with only 1.4T tokens, Xmodel-2.5 achieves 52.49% average accuracy across 13 reasoning benchmarks, ranking second among 1-2B parameter models, only behind Qwen3 (56.96%) but with 25.7x fewer training tokens.

Model Architecture

Intended Uses & Limitations

Intended Uses

• Complex reasoning tasks
• Lightweight AI agent applications
• Educational and research purposes
• Resource-constrained environments

Limitations

• Limited to 1.3B parameter capacity
• May struggle with highly specialized domains
• Performance may vary on non-English languages

Training Details

Training Strategy

• Three-stage WSD curriculum: 560k steps, 1.4T tokens
• Warmup phase: 2k steps, linear learning rate increase
• Stable phase: 530k steps, gradually increasing batch size
• Decay phase: 20k steps, mixing 66.9% high-quality SFT data
• Long-context adaptation: 10k additional steps for 16K context support

Key Innovations

• μP hyperparameter transfer: Direct transfer from 20M parameter proxy model to full model
• Optimizer switching: AdamW → Muon during decay phase for improved reasoning performance
• FP8 mixed precision: FP8 format significantly enhances training efficiency

Performance

Comprehensive Reasoning Performance

Detailed Task Performance

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import os

model_path = "XiaoduoAILab/Xmodel-2.5"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    model_path,
    trust_remote_code=True
)

prompt = "Explain the concept of transfer learning in machine learning."
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer(text, return_tensors="pt").to(model.device)

# Generation configuration
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512,
    do_sample=True,
    top_p=0.9,
    temperature=0.7,
    pad_token_id=tokenizer.eos_token_id
)

output = tokenizer.decode(
    generated_ids[0][len(model_inputs.input_ids[0]):], 
    skip_special_tokens=True
)
print("Generated Response:")
print(output)

Citation

If you find Xmodel-2.5 useful for your research or applications, please consider citing our work:

@misc{liu2025xmodel25,
      title={Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM}, 
      author={Yang Liu and Xiaolong Zhong and Ling Jiang},
      year={2025},
      eprint={2511.19496},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2511.19496}, 
}

Contact

For questions or suggestions, please contact us through:

• GitHub Issues: Xmodel-2.5 Issues
• Email: [email protected]

License

This project is licensed under the Apache-2.0 License. See the LICENSE file for details.