qwen3-4b-instruct-dpo-from-sft-4096

qwen3-4b-instruct-dpo-from-sft-4096

This repository provides a LoRA adapter trained with Direct Preference Optimization (DPO).

• Base model: Qwen/Qwen3-4B-Instruct-2507
• Initialized from SFT LoRA: /content/lora_sft_best
• Preference dataset: u-10bei/dpo-dataset-qwen-cot

This repo contains LoRA adapter weights only. Load it on top of the base model.

Training Objective

Align responses with preferred outputs using DPO (pairwise preference learning).

Training Configuration

• Method: DPO (TRL) + Unsloth (4-bit QLoRA base loading)
• Epochs: 1
• Learning rate: 5e-07
• Beta: 0.1
• Max sequence length: 1024
• Max prompt length: 512

Usage (Load LoRA)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_id = "Qwen/Qwen3-4B-Instruct-2507"
lora_id = "WassyO/qwen3-4b-instruct-dpo-from-sft-4096"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(base, lora_id)

prompt = "Your question here"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sources & Terms (IMPORTANT)

• Training data: u-10bei/dpo-dataset-qwen-cot
• Compliance: Users must follow the dataset terms and the base model license.