DeepSeek-R1-Distill-Qwen-1.5B-LIMO

Model Overview

DeepSeek-R1-Distill-Qwen-1.5B-LIMO is a fine-tuned version of DeepSeek-R1-Distill-Qwen-1.5B, trained on the LIMO (Less Is More for Reasoning) dataset. This fine-tune focuses on improving mathematical reasoning and problem-solving capabilities while maintaining efficient parameter scaling. The LIMO dataset, containing only 817 high-quality reasoning samples, challenges conventional scaling laws by demonstrating strong performance with minimal data.

Model Name: Josephgflowers/DeepSeek-R1-Distill-Qwen-1.5B-LIMO

---

Key Features

• Mathematical Reasoning Focus: Trained on the LIMO dataset to enhance step-by-step logical deduction and problem-solving.
• Optimized for Efficiency: Based on DeepSeek-R1-Distill-Qwen-1.5B, a smaller distilled model that retains powerful reasoning capabilities from its larger counterparts.
• Chain-of-Thought (CoT) Emphasis: Encourages structured, step-by-step reasoning with improved interpretability.
• Minimal Data, Strong Generalization: Leverages only 817 high-quality training samples to achieve competitive reasoning performance.

---

Model Details

• Base Model: DeepSeek-R1-Distill-Qwen-1.5B
• Parameter Count: 1.5B
• Training Framework: Unsloth / Hugging Face Transformers
• Dataset: LIMO (Less Is More for Reasoning)
• Primary Use Cases:
  • Mathematical and logical reasoning
  • STEM education and tutoring
  • Instruction-following for structured problem-solving

---

Training Data

This model was fine-tuned using the LIMO dataset, which emphasizes reasoning efficiency by achieving competitive performance with only 817 training samples.

Dataset Highlights:

• Name: LIMO (Less Is More for Reasoning)
• Size: 817 samples
• Focus: Chain-of-thought reasoning for structured problem-solving
• Key Motivation:
  • High-quality, curated reasoning samples outperform large-scale noisy data.
  • Designed to enhance reasoning generalization in a minimal-data setting.

---

Known Issues & Limitations

• System Instructions: The model may require explicit user instructions to enforce structured reasoning.
• Generalization Scope: While the LIMO dataset is effective for reasoning, additional fine-tuning may be required for broader NLP applications.
• Computation Constraints: As a 1.5B parameter model, it is more lightweight than larger reasoning models but still requires moderate computational resources.

@misc{ye2025limoreasoning, title={LIMO: Less is More for Reasoning}, author={Yixin Ye and Zhen Huang and Yang Xiao and Ethan Chern and Shijie Xia and Pengfei Liu}, year={2025}, eprint={2502.03387}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2502.03387}, }

@misc{deepseekai2025deepseekr1incentivizingreasoningcapability, title={DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning}, author={DeepSeek-AI and multiple contributors}, year={2025}, eprint={2501.12948}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2501.12948}, }