Downloads · 30 days
4
10% of all-time downloads
luckyman2907/lab22-dpo-qwen2.5-3b-vn
lab22-dpo-qwen2.5-3b-vn is a machine learning model from luckyman2907. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
This is a DPO (Direct Preference Optimization) LoRA adapter trained on top of an SFT checkpoint for Vietnamese chat alignment.
Downloads · 30 days
4
10% of all-time downloads
All-time downloads
42
Public
Repo size
131 MB
Likes
0
Public
Click a slice to open those files.
.safetensors120 MB · 91%
From the Hugging Face model README
This is a DPO (Direct Preference Optimization) LoRA adapter trained on top of an SFT checkpoint for Vietnamese chat alignment.
unsloth/Qwen2.5-3B-bnb-4bitThis adapter must be loaded on top of an SFT checkpoint. The training pipeline is:
from unsloth import FastLanguageModel
from peft import PeftModel
# Load base model
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen2.5-3B-bnb-4bit",
max_seq_length=512,
load_in_4bit=True,
)
# Load SFT adapter first (you need to have this from Lab 21)
model = PeftModel.from_pretrained(model, "path/to/sft-mini")
# Load DPO adapter on top
model = PeftModel.from_pretrained(model, "luckyman2907/lab22-dpo-qwen2.5-3b-vn")
# Prepare for inference
FastLanguageModel.for_inference(model)
# Generate
messages = [{"role": "user", "content": "Giải thích thuật toán quicksort"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")
outputs = model.generate(input_ids=inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-3B",
load_in_4bit=True,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B")
# Load SFT then DPO
model = PeftModel.from_pretrained(base_model, "path/to/sft-mini")
model = PeftModel.from_pretrained(model, "luckyman2907/lab22-dpo-qwen2.5-3b-vn")
Evaluated on 8 prompts (4 helpfulness + 4 safety) with side-by-side comparison.
See full evaluation results in the GitHub repository.
5CD-AI/Vietnamese-Multi-turn-Chat-Alpacaargilla/ultrafeedback-binarized-preferences-cleaned@misc{lab22-dpo-qwen2.5-vn,
author = {Luckyman2907},
title = {Lab22 DPO Alignment - Vietnamese Chat Model},
year = {2026},
publisher = {HuggingFace},
howpublished = {\url{https://huggingface.co/luckyman2907/lab22-dpo-qwen2.5-3b-vn}}
}
Apache 2.0 (inherits from Qwen2.5 base model)