qwen3-math-finetuned

📘 qwen3-math-finetuned

This model is a LoRA fine-tuned version of Qwen/Qwen3-0.6B, optimized for math-related tasks. It has been trained using the PEFT framework with LoRA adapters to improve efficiency while keeping the base model lightweight.

✅ Evaluation Result:

• Final Validation Loss: 1.0896

---

🔎 Model Description

• Base Model: Qwen/Qwen3-0.6B
• Fine-tuning Approach: LoRA (PEFT)
• Pipeline: Text Generation
• Objective: Improve mathematical reasoning and problem-solving capabilities.

This model leverages parameter-efficient fine-tuning, making it suitable for running on modest GPU resources while maintaining strong performance in math problem-solving tasks.

---

🚀 Intended Uses & Limitations

Intended Uses

• Solving mathematical word problems
• Assisting in step-by-step reasoning
• Generating math-focused educational content
• Serving as a base for further fine-tuning on structured reasoning tasks

Limitations

• May not generalize well outside of math-specific datasets
• Can still produce hallucinations for complex reasoning problems
• Evaluation was limited to a small test set

---

📊 Training & Evaluation Data

• Dataset: Custom math dataset (details not specified)
• Split: Train (7,000 samples), Test (1,000 samples)
• Preprocessing: Tokenization with base Qwen tokenizer
• Data Collator: Causal LM objective (no MLM)

---

⚙️ Training Procedure

Hyperparameters

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• optimizer: ADAMW_TORCH_FUSED (betas=(0.9,0.999), eps=1e-08)
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP (fp16)

📈 Training Results

---

🛠 Framework Versions

• PEFT: 0.17.1
• Transformers: 4.56.1
• PyTorch: 2.8.0+cu126
• Datasets: 4.0.0
• Tokenizers: 0.22.0

---

✨ This model is an early step toward building a lightweight Math Reasoning LLM. Further fine-tuning with larger and more diverse math datasets is encouraged for better results.

---