qwen3b-finetuned

Qwen2.5-3B-Instruct Fine-Tuned Model

📌 Model Overview

This repository contains a fine-tuned version of Qwen2.5-3B-Instruct using Unsloth. The model is optimized for multi-hop reasoning, scientific Q&A, and retrieval-augmented generation (RAG) with FAISS and BM25 retrieval.

• Base Model: Qwen2.5-3B-Instruct
• Fine-Tuning Framework: Unsloth
• Quantization: 4-bit GGUF & 16-bit versions available
• Training Methods: SFT (Supervised Fine-Tuning) + ORPO (Offline Reward Preference Optimization)

---

🔥 Fine-Tuning Details

1️⃣ Datasets Used

• HotpotQA: Multi-hop reasoning dataset
• Synthetic QA: Created using extracted document chunks
• BM25 & FAISS Retrieval: Used to retrieve relevant documents

2️⃣ Training Configuration

• LoRA Fine-Tuning: PEFT with Unsloth

• Hyperparameters:

  • r=16, lora_alpha=16, lora_dropout=0
  • gradient_accumulation_steps=4
  • max_seq_length=2048
  • learning_rate=2e-4
  • max_steps=200
  • optimizer=adamw_8bit

• RL Fine-Tuning (ORPO): Used for improving reasoning performance

---

📁 Files Included

• pytorch_model-00001-of-00002.bin - Model weights
• pytorch_model-00002-of-00002.bin
• pytorch_model.bin.index.json - Index of model checkpoints
• config.json - Model configuration
• tokenizer.json - Tokenizer configuration
• tokenizer_config.json
• merges.txt - BPE merge rules
• vocab.json - Token vocabulary
• special_tokens_map.json
• generation_config.json - Default generation settings
• unsloth.Q4_K_M.gguf - Quantized 4-bit version for Llama-CPP
• unsloth.F16.gguf - 16-bit version for full precision inference

---

🚀 Model Usage

Load Model in Python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "HasinduNimesh/qwen3b-finetuned"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

input_text = "Why is it necessary to filter out chain-of-thought outputs with mixed languages, long paragraphs, and code blocks?"
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_length=256)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Use with Llama-CPP (4-bit GGUF)

from llama_cpp import Llama

llm = Llama(model_path="unsloth.Q4_K_M.gguf", n_ctx=2048)
prompt = "Summarize the latest research on AI safety."
output = llm(prompt, max_tokens=200)
print(output["choices"][0]["text"])

---

🛠 Future Improvements

• Improve dataset diversity: Add more diverse reasoning datasets
• Optimize retrieval: Enhance FAISS & BM25 hybrid retrieval
• Expand RL fine-tuning: Improve reward models for ORPO

---

🛡️ License

This model is available under the Apache 2.0 License. Please follow Hugging Face’s guidelines for responsible AI usage.

---

🤝 Acknowledgements

• Unsloth: For efficient Qwen fine-tuning
• Hugging Face: Model hosting & dataset tools
• DeepSeek & Qwen Teams: For providing base models

---

📢 For issues or improvements, please open a discussion on Hugging Face! 🚀