qwen2.5-3b-reasoning-qlora

Qwen2.5-3B Reasoning (QLoRA 4-bit Adapter)

This repository contains the fine-tuned weights for Qwen/Qwen2.5-3B-Instruct trained to produce explicit step-by-step <think>...</think> Chain-of-Thought (CoT) reasoning.

Model Summary

• Fine-Tuning Methodology: Method 1: QLoRA (4-bit NF4 Quantization + 16-bit LoRA Adapters)
• Base Model: Qwen/Qwen2.5-3B-Instruct
• Trainable Parameters: 18.4 Million (0.59% trainable parameters)
• Dataset: FreedomIntelligence/medical-o1-reasoning-SFT (10,000 instructions)
• Hardware Acceleration: 2x NVIDIA H200 NVL GPUs (288 GB VRAM)

---

Empirical Benchmark Performance

---

Usage Instructions

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-3B-Instruct"
adapter_model_id = "DexterSptizu/qwen2.5-3b-reasoning-qlora"

# Load base model & tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Attach fine-tuned reasoning adapter
model = PeftModel.from_pretrained(base_model, adapter_model_id)

# Inference Example
prompt = "<|im_start|>system\nYou are a helpful reasoning assistant.<|im_end|>\n<|im_start|>user\nSolve step by step: What is 15 * 24?<|im_end|>\n<|im_start|>assistant\n<think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Citation & Acknowledgements

• Base model developed by Alibaba Cloud (Qwen Team).
• Fine-tuning executed on dual NVIDIA H200 NVL GPUs.