assignment4-qwen25-7b-dpo-adapter

Model Card: assignment4-qwen25-7b-dpo-adapter

This repository contains a PEFT LoRA adapter fine-tuned with Direct Preference Optimization (DPO) for DSAA6000Q Assignment 4.

Model Details

• Adapter repo: xypkent/assignment4-qwen25-7b-dpo-adapter
• Base model: Qwen/Qwen2.5-7B-Instruct
• Fine-tuning method: DPO + LoRA (PEFT)
• Language: English
• Use case: course assignment and alignment experiment

Training Data

• Preference dataset: xypkent/assignment4-lima-pairrm-pref
• Source: instructions sampled from GAIR/lima
• Construction:
  1. Sample 50 instructions
  2. Generate 5 responses per instruction with Qwen2.5-7B-Instruct
  3. Rank responses with PairRM
  4. Build (prompt, chosen, rejected) pairs

Training Procedure

• Algorithm: DPO
• Epochs: 1
• Batch size (per device): 1
• Gradient accumulation steps: 8
• Adapter type: LoRA (PEFT)
• Frameworks: transformers, trl, peft, torch

Evaluation

• Sampled 10 unseen instructions (not in training set)
• Compared outputs from:
  • original Qwen/Qwen2.5-7B-Instruct
  • DPO model (base + this adapter)
• Comparison columns:
  • instruction
  • original_completion
  • dpo_completion

Limitations

• Small assignment-scale preference dataset (50 prompts)
• Not suitable for production or safety-critical deployment
• Can inherit biases/errors from base model and data

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "xypkent/assignment4-qwen25-7b-dpo-adapter"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=dtype,
    device_map="auto" if torch.cuda.is_available() else None,
)
if not torch.cuda.is_available():
    base_model.to("cpu")

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

Links

• Preference dataset: https://huggingface.co/datasets/xypkent/assignment4-lima-pairrm-pref
• Adapter repo: https://huggingface.co/xypkent/assignment4-qwen25-7b-dpo-adapter