dpo-qwen

Qwen3-4B DPO (merged weights + adapter checkpoints)

This repository contains:

1. Full merged 16-bit weights at the repo root (ready to use with Transformers), and
2. LoRA adapter checkpoints under adapters/ (saved during training).

Training Objective

This model was optimized using Direct Preference Optimization (DPO) to prefer chosen outputs over rejected ones, focusing on structured response quality.

Training Configuration

• Base model: Qwen/Qwen3-4B-Instruct-2507
• Method: DPO
• Epochs: 1
• Learning rate: 1e-06
• Beta: 0.05
• Max sequence length: 1024

Usage (merged weights)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Gen-oze/dpo-qwen"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = "Your question here"
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))