Downloads · 30 days
12
16% of all-time downloads
yavuz-ai/qwen3-4b-dpo-ultrafeedback-lora
qwen3-4b-dpo-ultrafeedback-lora is a text generation model from yavuz-ai. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
LoRA DPO adapter for Qwen3-4B-Instruct-2507, trained on UltraFeedback. one leg of crossfamily-dpo: the same recipe run across two model families (Qwen + Mistral).
Downloads · 30 days
12
16% of all-time downloads
All-time downloads
76
Public
Repo size
144 MB
Likes
0
Public
Click a slice to open those files.
.safetensors132 MB · 92%
From the Hugging Face model README
LoRA DPO adapter for Qwen3-4B-Instruct-2507, trained on UltraFeedback. one leg of
crossfamily-dpo: the same recipe run across
two model families (Qwen + Mistral).
held-out preference accuracy, measured with the DPO implicit-reward metric (reference-relative, so length cancels). n = 500 UltraFeedback pairs, disjoint from training.
| metric | value |
|---|---|
| implicit-reward accuracy | 0.666 |
| old absolute-logprob metric (broken baseline) | 0.498 |
| length-normalized accuracy | 0.592 |
| mean implicit-reward margin | +6.30 |
the first eval read 0.498 and looked like a null. it was a measurement bug, not a training bug. it
scored absolute log-probability, but DPO optimizes the reference-relative implicit reward
logp_policy(y) - logp_ref(y). re-scoring the unchanged checkpoint with the correct metric moved it
0.498 -> 0.666 with no retraining. full teardown (incl. a length-confound hypothesis that did not
survive the data) in the repo's NOTES.md.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
model = PeftModel.from_pretrained(base, "yavuz-ai/qwen3-4b-dpo-ultrafeedback-lora")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
hf jobs uv run src/train_dpo.py --flavor a100-large --secrets HF_TOKEN \
--env BASE_MODEL=Qwen/Qwen3-4B-Instruct-2507 \
--env OUTPUT_REPO=yavuz-ai/qwen3-4b-dpo-ultrafeedback-lora
the base is already preference-aligned, so this is a clean reference-relative preference lift on held-out data, not a state-of-the-art model. the value is the method and the failure analysis, reproduced identically on a Mistral model: ministral-8b-dpo-ultrafeedback-lora.