Downloads · 30 days
0
div1010/smollm2-rag-faithfulness-dpo
smollm2-rag-faithfulness-dpo is a machine learning model from div1010. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
QLoRA adapter trained with Direct Preference Optimization (DPO) to prefer context-grounded answers over plausible-sounding hallucinations in a RAG-style QA setting.
Downloads · 30 days
0
Access
Public
Updated Aug 8, 2026
Repo size
25.2 MB
Likes
0
Public
Click a slice to open those files.
.safetensors25.2 MB · 84%
From the Hugging Face model README
QLoRA adapter trained with Direct Preference Optimization (DPO) to prefer context-grounded answers over plausible-sounding hallucinations in a RAG-style QA setting.
This repo hosts the v2 adapter weights (lr=1e-5, beta=0.3). As detailed below, neither v1 nor v2 produced a verified faithfulness improvement — this project is shared primarily as a diagnosed negative result and a worked example of why reward/loss curves alone are insufficient to validate a DPO run. See "How to Improve This" for the config likely to work.
| Version | LR | Beta | Epochs | Faithfulness | Outcome |
|---|---|---|---|---|---|
| v1 | 5e-5 | 0.1 | 1 | 0.1342 (baseline: 0.2113) | Reward over-optimization — margin spiked 0.32→1.84 in 42 steps; model became less grounded, more hallucinated confidently |
| v2 (this repo) | 1e-5 | 0.3 | 1 | 0.2353 | Under-optimization — 45% of outputs identical to baseline; apparent gain is not real grounding improvement |
| v3 (not yet run) | 2e-5–3e-5 | 0.15–0.2 | 2 | — | Proposed bracket between v1/v2; see below |
~340 preference pairs built from SQuAD:
Two hyperparameter configs were run and diagnosed against each other — this is the actual finding of the project, not just a single before/after number.
| Config | LR | Beta | Faithfulness | Degenerate outputs | Identical-to-baseline |
|---|---|---|---|---|---|
| Baseline (no DPO) | — | — | 0.2113 | 7/60 (11.7%) | — |
| v1 | 5e-5 | 0.1 | 0.1342 (−0.077) | 11/60 (18.3%) | — |
| v2 | 1e-5 | 0.3 | 0.2353 (+0.024) | 9/60 (15.0%) | 27/60 (45.0%) |
v1 (lr=5e-5, beta=0.1): reward over-optimization. Reward margin climbed sharply during training (0.32 → 1.84 over 42 steps) while eval faithfulness dropped. Filtering out degenerate (near-empty) outputs from both baseline and v1 barely changed the gap (−0.0755 filtered vs. −0.0771 unfiltered), confirming this wasn't just empty-output noise — the model became genuinely less grounded even on answers it completed normally (e.g. reattributing an unrelated date from earlier in the context to the wrong entity).
v2 (lr=1e-5, beta=0.3): under-optimization. The apparent +0.024 improvement is misleading: 45% of v2's outputs are word-for-word identical to the untrained baseline, meaning the model barely moved from the reference policy. The small aggregate gain is mostly explained by a slightly lower degenerate-output rate (9/60 vs. baseline's 7/60 is actually still worse; v1's 11/60 was worse still), not by any real gain in grounding.
Neither config produced a genuine, verified improvement in faithfulness. The training loss and reward accuracy metrics alone (which looked good in both runs) would have been misleading without the qualitative and identical-output checks — this is the main methodological takeaway of the project.
Scored with cross-encoder/nli-deberta-v3-base on a held-out 15% split (not seen during training).
Ranked by expected impact, based on what the v1/v2 diagnosis revealed:
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-1.7B-Instruct")
model = PeftModel.from_pretrained(base, "div1010/smollm2-rag-faithfulness-dpo")
tokenizer = AutoTokenizer.from_pretrained("div1010/smollm2-rag-faithfulness-dpo")