lab22-dpo-vn

Lab 22 DPO Vietnamese Adapter

This is a PEFT/LoRA DPO adapter trained for the VinUni Day 22 DPO/ORPO Alignment Lab.

Base model

• Base model: unsloth/Qwen2.5-3B-bnb-4bit
• Compute tier: T4
• Adapter type: PEFT LoRA
• Training objective: Direct Preference Optimization

Training data

• SFT dataset: bkai-foundation-models/vi-alpaca
• SFT slice: 1,000 samples
• Preference dataset: argilla/ultrafeedback-binarized-preferences-cleaned
• Preference slice: 2,000 pairs on T4
• Preference columns: prompt, chosen, rejected

Hyperparameters

Evaluation summary

8-prompt side-by-side judge summary

Raw judge counts from data/eval/judge_results.json:

{
  "tie": 8
}

Benchmark results

How to load

from unsloth import FastLanguageModel
from peft import PeftModel

base_model = "unsloth/Qwen2.5-3B-bnb-4bit"
adapter_path = "solar11781/lab22-dpo-vn"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=base_model,
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
)

model = PeftModel.from_pretrained(model, adapter_path)

Intended use

This adapter is a student lab artifact for studying SFT-to-DPO alignment behavior, reward curves, qualitative helpfulness/safety comparison, and benchmark trade-offs.

Limitations

This is not a production assistant. It was trained on a small SFT slice and a small preference slice. Outputs may be incorrect, unsafe, overly brief, or misaligned. Use only for educational evaluation.