Downloads · 30 days
5
12% of all-time downloads
lab22-dpo-vn
TunaMonsieur/lab22-dpo-vn
lab22-dpo-vn is a machine learning model from TunaMonsieur. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter trained with DPO (TRL) on an SFT-mini checkpoint. VinUni AICB Track-3 Day-22.
All-time downloads
43
Public
Repo size
131 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors120 MB · 88%
At a glance
- Library
- peft
- License
- apache-2.0
- Access
- Public
- Created
- Jun 26, 2026
- Updated
- Jun 26, 2026
- SHA
- 19ad34ac
Base models
Model card
From the Hugging Face model README
lab22-dpo-vn — DPO-aligned Qwen2.5-3B (VN)
LoRA adapter trained with DPO (TRL) on an SFT-mini checkpoint. VinUni AICB Track-3 Day-22.
Hyperparameters
- Base: unsloth/Qwen2.5-3B-bnb-4bit (NF4 4-bit)
- Method: DPO, LoRA r=16, alpha=32
- beta=0.1, lr=5e-7, 1 epoch, 2000 UltraFeedback pairs, max_length=512
- Compute: free Colab T4
Results
- End reward gap (chosen - rejected): +0.0436
- Final chosen / rejected reward: -0.546 / -0.590
- NB4 (8 prompts, gpt-4o-mini judge): DPO 1 win / 7 tie -> win-rate 0.5625
- Note: reward gap positive but small; chosen reward did not rise, gap grew because rejected fell more (likelihood displacement). DPO on UltraFeedback (helpfulness) did not improve safety refusals.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "unsloth/Qwen2.5-3B-bnb-4bit"
tok = AutoTokenizer.from_pretrained(base)
m = AutoModelForCausalLM.from_pretrained(base, load_in_4bit=True, device_map="auto")
m = PeftModel.from_pretrained(m, "TunaMonsieur/lab22-dpo-vn")
- Library
- peft
- License
- apache-2.0
- Languages
- vi, en
- Created
- Jun 26, 2026
- Updated
- Jun 26, 2026