dpo-qwen-cot-merged_3

qwen3-4b-structured-output-lora

This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) via the Unsloth library.

This repository contains LoRA adapter weights only. The base model must be loaded separately.

Training Objective

This adapter is trained to improve structured output accuracy (JSON / YAML / XML / TOML / CSV).

Loss is applied only to the final assistant output, while intermediate reasoning (Chain-of-Thought) is masked.

In addition, it has been optimized using DPO to align its responses with preferred outputs, focusing on improving reasoning (Chain-of-Thought) and structured response quality based on the provided preference dataset.

Training Configuration

• Base model: Qwen/Qwen3-4B-Instruct-2507

1. Supervised Fine-Tuning (SFT)

• Method: QLoRA (4-bit)
• Max sequence length: 512
• Epochs: 1
• Learning rate: 2e-05
• LoRA: r=64, alpha=128

2.Direct Preference Optimization (DPO)

• Epochs: 1
• Learning rate: 1e-07
• Beta: 0.1
• Max sequence length: 1024
• LoRA: r=8, alpha=16

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

model_id = "SusumuDou/dpo-qwen-cot-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter)

# Test inference
prompt = "Your question here"
inputs = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))

Sources & License (IMPORTANT)

• Training Data: SFT -- [u-10bei/structured_data_with_cot_dataset_512_v2] DPO -- [u-10bei/dpo-dataset-qwen-cot]
• License: MIT License. (As per dataset terms).
• Compliance: Users must follow the original base model's license terms.