Downloads · 30 days
2
3% of all-time downloads
yavuz-ai/ministral-8b-dpo-ultrafeedback-lora
ministral-8b-dpo-ultrafeedback-lora is a text generation model from yavuz-ai. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as other.
LoRA DPO adapter for Ministral-8B-Instruct-2410, trained on UltraFeedback. one leg of crossfamily-dpo: the same recipe run across two model families (Qwen + Mistral).
Downloads · 30 days
2
3% of all-time downloads
All-time downloads
59
Public
Repo size
192 MB
Likes
0
Public
Click a slice to open those files.
.safetensors175 MB · 91%
From the Hugging Face model README
LoRA DPO adapter for Ministral-8B-Instruct-2410, trained on UltraFeedback. one leg of
crossfamily-dpo: the same recipe run across
two model families (Qwen + Mistral).
inherits the Mistral Research License (research / non-commercial) from the base model.
held-out preference accuracy, measured with the DPO implicit-reward metric (reference-relative, so length cancels). n = 500 UltraFeedback pairs, disjoint from training.
| metric | value |
|---|---|
| implicit-reward accuracy | 0.672 |
| old absolute-logprob metric (broken baseline) | 0.514 |
| length-normalized accuracy | 0.640 |
| mean implicit-reward margin | +6.53 |
same recipe on Qwen3-4B-Instruct lands 0.666, so the effect holds across families.
the first run of this pipeline (on the Qwen leg) read 0.498 and looked like a null. it was a
measurement bug, not a training bug: it scored absolute log-probability, but DPO optimizes the
reference-relative implicit reward logp_policy(y) - logp_ref(y). the corrected metric is what is
reported above. full teardown in the repo's NOTES.md.
mistral class; verified all weight shards load with
no re-initialization, so the warning is a benign config-type alias.from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("mistralai/Ministral-8B-Instruct-2410")
model = PeftModel.from_pretrained(base, "yavuz-ai/ministral-8b-dpo-ultrafeedback-lora")
tok = AutoTokenizer.from_pretrained("mistralai/Ministral-8B-Instruct-2410")
hf jobs uv run src/train_dpo.py --flavor a100-large --secrets HF_TOKEN \
--env BASE_MODEL=mistralai/Ministral-8B-Instruct-2410 \
--env OUTPUT_REPO=yavuz-ai/ministral-8b-dpo-ultrafeedback-lora
the base is already preference-aligned, so this is a clean reference-relative preference lift on held-out data, not a state-of-the-art model. the value is the method and the failure analysis, reproduced identically on a Qwen model: qwen3-4b-dpo-ultrafeedback-lora.