dpo-qwen2.5-7b-adapter

DPO LoRA Adapter — Qwen2.5-7B-Instruct (beta=0.3)

A LoRA adapter trained with Direct Preference Optimization (DPO) on top of Qwen/Qwen2.5-7B-Instruct, on a subset of UltraFeedback. Trained under a tight compute budget (single L40S, 2–3 hr window) as part of a two-point beta ablation study; this checkpoint is the beta=0.3 run.

Model Details

• Base model: Qwen/Qwen2.5-7B-Instruct
• Method: DPO (Direct Preference Optimization), LoRA adapter only (base weights not modified)
• Training data: 4,000-example subset of UltraFeedback (binarized)
• Hyperparameters: beta=0.3, 1 epoch, effective batch size 16 (per-device batch 1, grad accumulation 16)
• Hardware: 1x L40S GPU, ~35–45 min training time
• Framework: TRL DPOTrainer + PEFT LoRA

Evaluation

Evaluated with a position-bias-controlled pairwise LLM judge (Llama-3.3-70B via Groq API) on 150 held-out UltraFeedback prompts, comparing this DPO-adapted model against the unmodified Qwen2.5-7B-Instruct baseline. Each pair was judged twice with response order swapped to detect position bias.

Honest interpretation: the ~90% tie/position-disagreement rate indicates the judge could not reliably discriminate between baseline and DPO outputs for most prompts under this training budget. This is consistent with a parallel run at beta=0.1 on a different base model (Llama-3.1-8B-Instruct), which showed a near-identical ~89% instability rate. Loss and reward margins moved in the expected direction during training, so this reflects a measurement/training-budget limitation (small dataset, single epoch, possibly judge-capacity limits at 8B-class), not a training failure.

Intended Use

This is a research/portfolio artifact demonstrating a DPO training and rigorous pairwise-eval pipeline under constrained compute — not intended for production deployment.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "svjay/dpo-qwen2.5-7b-adapter")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

Limitations

• Trained on a small (4,000-example) subset for 1 epoch due to compute constraints.
• Judge model is a different family from both base models; absolute judge reliability at this scale is untested against human preference data.
• No safety-specific tuning or evaluation was performed.