lab22-dpo-vn

Lab 22 — DPO-Aligned Qwen2.5-3B (VN)

Vietnamese-aligned model produced by VinUniversity AICB Day-22 lab (Track 3 — DPO/ORPO Alignment).

Pipeline: SFT (1k VN Alpaca) → DPO (2k UltraFeedback, β=0.1, lr=5e-7) on top of unsloth/Qwen2.5-3B-bnb-4bit.

Files

• adapter_config.json + adapter_model.safetensors — DPO LoRA (rank=16, α=32). Stack on top of the base + SFT-mini adapter.
• lab22-dpo-Q4_K_M.gguf — merged + quantized GGUF for llama.cpp / llama-cpp-python deployment (~1.9 GB).

Quick start

Inference via llama-cpp-python

from llama_cpp import Llama
llm = Llama(model_path="lab22-dpo-Q4_K_M.gguf", n_ctx=512)
print(llm.create_chat_completion(messages=[{"role": "user", "content": "Giải thích quicksort."}])
        ["choices"][0]["message"]["content"])

Inference via transformers + PEFT

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B", torch_dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(base, "hiepphambk/lab22-dpo-vn")  # this DPO adapter

Training details

Evaluation results

See full report (incl. reward curves analysis, alignment-tax interpretation, and W&B run link) in the lab repo.

Caveats

• Trained on English UltraFeedback pref data — VN behavior improves via transfer; native-VN pref dataset would be better (deck §5.4).
• 3B + 1k SFT + 2k DPO is demonstrative scale, not production-ready. For production, use ≥ 7B base + ≥ 50k pref pairs.
• Likelihood displacement observed (deck §3.4): both chosen and rejected reward decrease, gap widens because rejected falls faster.

Citation / acknowledgements

• Lab template: VinUniversity AICB Day-22 (Track 3, A20 cohort 2026).
• Stack: Unsloth, TRL, PEFT, bitsandbytes, llama.cpp, lm-eval-harness.

Trained by Phạm Hữu Hoàng Hiệp (MSSV 2A202600415).