rag-sft-qwen2.5-0.5b

Qwen2.5-0.5B-Instruct QLoRA SFT (arXiv QA)

QLoRA-SFT fine-tune of Qwen/Qwen2.5-0.5B-Instruct on 150 (question, answer) pairs generated from a corpus of ML arXiv paper abstracts. LoRA rank 16, alpha 32, targeting the attention projection layers, 4-bit NF4 quantized base model, 3 epochs.

Part of a larger RAG + GPT-from-scratch + DPO alignment portfolio project: https://github.com/sidesshmore/rag-document-assistant

Training

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Data: 150 QA pairs generated via qwen3-235b-a22b-instruct-2507 from sampled corpus chunks (135 train / 15 eval split)
• Method: QLoRA (4-bit NF4), LoRA r=16, alpha=32, dropout=0.05, targeting q_proj, k_proj, v_proj, o_proj
• Result: eval loss 2.486 -> 2.134 -> 2.047 over 3 epochs, mean token accuracy 0.56 -> 0.61

Downstream use

This checkpoint is the starting point for a DPO alignment pass - see sidmhuggingface/rag-dpo-qwen2.5-0.5b for the DPO-aligned version and an honest comparison. With bias-controlled pairwise judging (independent judge model, position-swap reconciliation), DPO did not clearly beat this SFT checkpoint on held-out questions at this data scale (40 preference pairs) - see that model's card for the full analysis, since that's arguably the more interesting result.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base, "sidmhuggingface/rag-sft-qwen2.5-0.5b")
tokenizer = AutoTokenizer.from_pretrained("sidmhuggingface/rag-sft-qwen2.5-0.5b")

Framework versions

• PEFT 0.19.1
• TRL 0.24.0
• Transformers 5.12.1
• PyTorch 2.11.0