rag-dpo-qwen2.5-0.5b

Qwen2.5-0.5B-Instruct DPO-aligned (arXiv QA)

DPO alignment of sidmhuggingface/rag-sft-qwen2.5-0.5b (itself a QLoRA-SFT of Qwen/Qwen2.5-0.5B-Instruct) on 40 preference pairs: chosen = qwen3-235b-a22b-instruct-2507 (a much larger model) answers, rejected = the SFT model's own generations on the same questions.

Part of a larger RAG + GPT-from-scratch + DPO alignment portfolio project: https://github.com/sidesshmore/rag-document-assistant

Training

• Method: DPO (trl DPOTrainer), beta=0.1, 3 epochs, LoRA r=16 continuing from the SFT adapter
• Data: 40 preference pairs (34 train / 6 eval), questions distinct from the SFT training set
• Training-time signal: reward accuracy 0.79 -> 0.94 -> 1.0 across 3 epochs, reward margin 0.37 -> 1.23, eval loss 0.39 -> 0.32 - DPO clearly learned to separate chosen from rejected during training.

Honest evaluation: DPO did not clearly beat SFT on held-out generation quality

Pairwise win rate on 20 held-out questions, judged with gemma3-27b-it - deliberately a different model family from the qwen3-235b DPO teacher, to avoid the judge favoring outputs that resemble its own family. Each pair was judged twice with (SFT, DPO) presentation order swapped and reconciled to a winner or "tie" - a standard control for LLM-judge position bias.

SFT alone won more often than the DPO-aligned model. This is reported as-is, not tuned away: an earlier check with a different judge (before the bias controls above were added, and before that run hit an unrelated API error) showed the same mixed pattern, which cross-checks that this isn't a one-off artifact of the specific judge model.

The likely explanation is that the training signal here is thin by construction: a 0.5B parameter model, LoRA rank 16, and only 40 preference pairs. The gap is between "the model learned to prefer chosen over rejected during training" (which the reward-accuracy curve above shows clearly) and "that generalizes to better held-out answers judged independently" (which it does not, at this scale). A larger preference set or more epochs might change this - untested here, and worth flagging as a natural next step rather than retroactively "fixing" the number by re-running until it looks better.

Sample generations (held-out questions)

Q: What is the significance of treewidth in determining the efficiency of exact inference in graphical models?

• SFT: "Treewidth plays a crucial role in evaluating the complexity of exact inference for graphical models. A model with low treewidth can be more efficiently solved using approximate algorithms..."
• DPO: "Treewidth plays a crucial role in assessing the complexity of exact inference for graphical models. A model with low treewidth (e.g., 2 or less) generally has simpler structures..."
• Judge verdict: SFT

Q: What method was used to generate in-distribution adversarial attacks in the study on CNN robustness for lumbar disk shape reconstruction?

• SFT: "The study generated in-distribution adversarial attacks using a combination of adversarial training and data augmentation techniques."
• DPO: "The study utilized a combination of adversarial training and in-distribution adversarial attacks to improve the robustness of CNNs for lumbar disk shape reconstruction."
• Judge verdict: tie

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base, "sidmhuggingface/rag-dpo-qwen2.5-0.5b")
tokenizer = AutoTokenizer.from_pretrained("sidmhuggingface/rag-dpo-qwen2.5-0.5b")

Framework versions

• PEFT 0.19.1
• TRL 0.24.0
• Transformers 5.12.1
• PyTorch 2.11.0

Citation

@inproceedings{rafailov2023direct,
    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
    year         = 2023,
    booktitle    = {Advances in Neural Information Processing Systems 36},
}