lab22-dpo-adapter

Lab 22 Vietnamese Qwen2.5-3B DPO Adapter

This repository contains the final LoRA adapter from a small Direct Preference Optimization (DPO) experiment for VinUni AICB Track 3, Day 22. It is an adapter only, not a standalone base model. Load it on top of unsloth/Qwen2.5-3B-bnb-4bit (or a compatible Qwen2.5-3B base).

Intended use and limitations

The adapter is a course-lab artifact for Vietnamese instruction-following experiments. It is not validated for production use, high-stakes decisions, or safety-critical deployment. In the small qualitative evaluation, DPO won 2 of 8 comparisons, SFT won 1, and 5 were ties. The four safety prompts were ties; the adapter must not be treated as a reliable safety system.

Training

• Base model: unsloth/Qwen2.5-3B-bnb-4bit
• Hardware: Kaggle Tesla T4
• Preference data: 2,000 pairs from argilla/ultrafeedback-binarized-preferences-cleaned
• Objective: DPO, one epoch, LoRA rank 16 / alpha 32
• Final DPO loss: 0.7875
• Final chosen-minus-rejected reward gap: +0.1436

The original SFT warm-start used 1,000 samples from bkai-foundation-models/vi-alpaca. PyTorch SDPA was used because the available Tesla T4/xFormers combination lacked a compatible backward kernel.

Evaluation

The course qualitative evaluation used eight Vietnamese prompts and an OpenAI-compatible gpt-4o-mini judge. Results: DPO 2 wins, SFT 1 win, and 5 ties. No broad benchmark scores are claimed because NB6 was not completed.

Loading the adapter

from unsloth import FastLanguageModel
from peft import PeftModel

base, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-3B-bnb-4bit",
    max_seq_length=512,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(base, "YOUR_HF_USERNAME/lab22-dpo-adapter")
FastLanguageModel.for_inference(model)

License and attribution

This course artifact inherits the usage terms of its base model and source datasets. Do not use it to generate harmful, illegal, or unsafe instructions.