Downloads · 30 days
5
8% of all-time downloads
decor-qwen3-4b-original
Ciaranshu/decor-qwen3-4b-original
decor-qwen3-4b-original is a text generation model from Ciaranshu. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is Qwen3-4B fine-tuned on the original (unmodified) LIMO dataset. It serves as the baseline for comparison with DSE-cleaned variants.
All-time downloads
65
Public
Parameters
4.4B
8.8 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8.8 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- qwen3
- Access
- Public
- Created
- Mar 2, 2026
- Updated
- Mar 2, 2026
- SHA
- 3b6da007
Try a prompt
Base models
Model card
From the Hugging Face model README
DecoR: Qwen3-4B Fine-tuned on Original LIMO (Baseline)
This model is Qwen3-4B fine-tuned on the original (unmodified) LIMO dataset. It serves as the baseline for comparison with DSE-cleaned variants.
Key Results
| Benchmark | Base Qwen3-4B | Original LIMO SFT | DSE LIMO SFT |
|---|---|---|---|
| MATH-500 | 56.6% | 69.6% | 72.8% |
| AIME 2025 | 13.3% | 40.0% | 43.3% |
| AIME 2026 | 36.7% | 46.7% | 53.3% |
| GPQA Diamond | 43.4% | 55.6% | 49.0% |
Training Details
- Base model: Qwen/Qwen3-4B
- Training data: LIMO-Original (817 samples, unmodified)
- Framework: LLaMA-Factory + DeepSpeed ZeRO-2
- Hardware: 2× NVIDIA A100 80GB (CSD3 HPC)
- Hyperparameters:
- Learning rate: 5e-6 (cosine schedule, 10% warmup)
- Batch size: 8 (effective)
- Epochs: 15
- Max sequence length: 16384
- Full fine-tuning (no LoRA)
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Ciaranshu/decor-qwen3-4b-original")
tokenizer = AutoTokenizer.from_pretrained("Ciaranshu/decor-qwen3-4b-original")
Framework versions
- Transformers 4.52.4
- Pytorch 2.5.1+cu124
- Datasets 3.6.0
- Tokenizers 0.21.1
- Task
- Text Generation
- Library
- transformers
- Type
- qwen3
- License
- apache-2.0
- Created
- Mar 2, 2026
- Updated
- Mar 2, 2026