lab22-dpo-qwen2.5

Qwen2.5-0.5B DPO Vietnamese Alignment (MLX)

This model is a fine-tuned version of Qwen/Qwen2.5-0.5B aligned using Direct Preference Optimization (DPO) on a Vietnamese preference dataset. This was developed as part of Lab 22: DPO Alignment on Apple Silicon using the MLX framework.

Model Details

• Developed by: Do Hai Nam (VinUni)
• Model type: Causal Language Model (LoRA + DPO)
• Language(s): Vietnamese, English
• Finetuned from model: Qwen/Qwen2.5-0.5B
• Framework: MLX (Metal Performance Shaders)
• Hardware: Apple M2 Pro (16GB Unified Memory)

Uses

This model is intended for research purposes in LLM alignment and Vietnamese language processing. It is optimized for running locally on Apple Silicon devices.

How to use with MLX-LM

from mlx_lm import load, generate

model, tokenizer = load("baeGil/lab22-dpo-qwen2.5")
response = generate(model, tokenizer, prompt="Thủ đô của Việt Nam là gì?", verbose=True)

Training Details

Training Data

• SFT Stage: Viet-AI/vietnamese-alpaca (1,000 samples)
• DPO Stage: argilla/ultrafeedback-binarized-preferences-cleaned (1,800 pairs, Vietnamese context)

Training Procedure

• Method: LoRA + DPO (Direct Preference Optimization)
• Hyperparameters:
  • Beta: 0.1
  • Learning Rate: 2e-7
  • LoRA Rank (r): 16
  • LoRA Alpha: 32
  • Precision: Float16 (FP16)
  • Optimizer: Adam

Evaluation

Results

• Reward Gap (Chosen - Rejected): Increased from -0.19 to +0.146 over training.
• Qualitative: Significant reduction in repetitive text compared to the SFT-only baseline.

Technical Specifications

The model is released in two formats:

1. MLX LoRA Adapter: For use with the mlx-lm library.
2. GGUF (Q4_K_M): For use with llama.cpp and other GGUF-compatible runners.

Citation

If you use this model, please credit the Lab 22 DPO Alignment curriculum and the MLX framework.