dpo-implicit-reward

Exploring DPO's Implicit Reward

Research checkpoints for a controlled study asking whether DPO's implicit reward

r̂(x, y) = β · [ log π_θ(y|x) − log π_ref(y|x) ]

works as well as an explicitly trained Bradley–Terry reward model, under matched data, in-distribution and under distribution shift.

• H1: explicit RM > implicit on held-out in-distribution pairs. → supported
• H2: the gap widens under distribution shift. → not supported

Code: github.com/silvererudite/dpo-implicit-reward

---

⚠️ Read this before using any checkpoint

π_ref must be sft_merged/. Every adapter here was trained on top of the merged SFT checkpoint, not on raw Qwen/Qwen2.5-0.5B. Loading an adapter onto the raw base model gives a different — and wrong — model. For the implicit reward specifically, using the wrong reference silently changes the quantity you compute.

These are research artifacts at 0.5B scale, not production reward models. They are not safety-aligned and should not be used to rank or moderate content in a deployed setting.

---

What is in this repository

Reference policy — start here

Main experiment — 1 epoch, 5 seeds

The headline result. Each run saw exactly 8,000 preference pairs in 500 optimizer steps.

Convergence experiment — 4 epochs, 2 seeds

Trained to convergence with held-out evaluation every 100 steps, to test whether the 1-epoch ranking was an artifact of stopping early. It was not.

Results and figures

Every training directory carries a budget.json recording steps, epochs, effective batch, LR, seed and wall-clock, so the compute-matching claim is auditable rather than asserted.

---

How to load each artifact

1. The reference policy π_ref

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "Shamima/dpo-implicit-reward"
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

# subfolder= pulls just this directory out of the repo
ref = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="sft_merged", torch_dtype=torch.bfloat16).cuda().eval()

2. A DPO policy → the implicit reward

from peft import PeftModel

backbone = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="sft_merged", torch_dtype=torch.bfloat16).cuda()
policy = PeftModel.from_pretrained(backbone, REPO, subfolder="dpo_beta0.1_8k_s0").cuda().eval()

# r̂ = β · (log π_θ(y|x) − log π_ref(y|x)), summed over RESPONSE tokens only
# (prompt masked). See src/scoring.py in the GitHub repo for the exact implementation.

The reward is a difference of two models, so you need both policy and ref loaded. Ranking within a prompt is invariant to β, so β only affects calibration, not pairwise accuracy.

3. An explicit reward model

from transformers import AutoModelForSequenceClassification

base = AutoModelForSequenceClassification.from_pretrained(
    REPO, subfolder="sft_merged", num_labels=1, torch_dtype=torch.bfloat16)
base.config.pad_token_id = tok.pad_token_id
rm = PeftModel.from_pretrained(base, REPO, subfolder="rm_8k_s0").cuda().eval()

text = tok.apply_chat_template(
    [{"role": "user", "content": prompt},
     {"role": "assistant", "content": response}], tokenize=False)
score = rm(**tok(text, return_tensors="pt", truncation=True, max_length=1024).to("cuda")).logits[0].item()

The trained scalar head ships inside the adapter under modules_to_save, so loading the adapter restores it. Transformers will warn that score.weight was "newly initialized" when the base is created — that warning is expected and the adapter overwrites it.

4. A mid-training checkpoint

rm_at_epoch2 = PeftModel.from_pretrained(
    base, REPO, subfolder="long/rm_8k_s0/checkpoint-1000").cuda().eval()

Step → epoch mapping: 500 steps = 1 epoch (8,000 pairs at effective batch 16). So checkpoint-250 = 0.5 epochs, checkpoint-1000 = 2 epochs, checkpoint-2000 = 4 epochs.

5. Results without any GPU

from huggingface_hub import hf_hub_download
import json

p = hf_hub_download(REPO, "results/aggregate_beta0.1_8k.json", repo_type="model")
agg = json.load(open(p))["results"]

---

Which checkpoint should I use?

Do not use long/*/checkpoint-2000 (4 epochs) expecting the best model — both scorers are past their peak there. The reward model's held-out accuracy falls from 0.731 at epoch 2 to ~0.69 at epoch 4.

---

Results

Length-controlled pairwise accuracy, mean ± 95% CI over 5 seeds. Length control restricts scoring to pairs whose two responses are within 1.2× in length, because raw accuracy on these benchmarks is substantially a length signal.

The "untrained baseline" is the length-normalized log-probability of the response under π_ref — no reward training at all.

Three findings

1. H1 holds. The explicit reward model wins in distribution, +0.062 ± 0.030, in every seed, and the effect survives correction for the six simultaneous comparisons.
2. H2 does not hold. Four of five shifted sets are ties at 5 seeds. The one clearly resolvable OOD difference runs the other way (Reasoning).
3. On the two hardest sets, reward training is worse than no reward training. The untrained baseline beats both trained scorers on Reasoning and Chat-Hard. Whatever these methods learn, they also destroy signal the base model already had.

Convergence and drift

Training both to 4 epochs: the reward model peaks at 0.731 held-out accuracy at epoch 2 then overfits; DPO peaks at 0.676 at epoch 1 and is flat thereafter. The ranking never inverts.

Scoring every checkpoint for surface-form preference produced a result that refuted our hypothesis: we expected the reward model's length bias to grow with training. It peaks at +0.069 around epoch 1.5 and then fades to +0.002. DPO's drifts the other way, from −0.134 to −0.232 — it progressively prefers shorter answers than humans do, and its accuracy falls with it.

---

Training setup

Qwen/Qwen2.5-0.5B, LoRA r=16 / α=32 / dropout=0.05 on all attention and MLP projections, bf16, single NVIDIA A10G per job, TRL 0.15.2. DPO and the reward model both initialize from sft_merged/ and see byte-identical pairs (verified by hash) so neither is advantaged.

Limitations

• 0.5B scale, one β, one data budget. Do not generalize to 7B+ without checking.
• π_ref was trained once. Seeds vary only the second stage, so the intervals describe training variance given one reference, not variance of the method. The untrained baseline consequently has no error bars at all.
• Length control costs power. Three test sets retain under 100 pairs after filtering and cannot support conclusions on their own.
• Calibration is poor for every scorer (ECE 0.22–0.45), and degrades faster than accuracy during overfitting.
• Compute is data-matched, not FLOP-matched. DPO runs an extra frozen reference forward pass and took ~1.7× the wall-clock.

License

Apache 2.0, following the Qwen/Qwen2.5-0.5B base model.