Qwen4b-SFT-d9-merged-after-dpo-d2

Qwen3-4B SFT + General DPO (Round 1)

This model is a fine-tuned version of Rakushaking/Qwen4b-SFT-d9 using Direct Preference Optimization (DPO) via the Unsloth library.

This repository contains the full-merged 16-bit weights. No adapter loading is required.

Training Objective

This model has been optimized using DPO with a general-purpose preference dataset (u-10bei/dpo-dataset-qwen-cot) to improve overall Chain-of-Thought reasoning quality and structured output consistency. This round focuses on broad output quality improvement rather than format-specific corrections.

Training Pipeline

1. Base: Qwen/Qwen3-4B-Instruct-2507
2. SFT: Structured data generation/conversion with Chain-of-Thought (V4+V5, ~9.3k samples)
3. DPO Round 1 (this model): Generic preference optimization (u-10bei/dpo-dataset-qwen-cot, ~4,040 pairs)

DPO Data

• Dataset: u-10bei/dpo-dataset-qwen-cot
• Pairs: ~4,040 preference pairs
• Content: General Chain-of-Thought quality preferences — chosen responses exhibit better reasoning structure and output accuracy compared to rejected responses

Training Configuration

• Base model: Rakushaking/Qwen4b-SFT-d9-merged
• Method: DPO (Direct Preference Optimization)
• Epochs: 1
• Learning rate: 5e-07
• Beta: 0.1
• Max sequence length: 1024
• Max prompt length: 512
• LoRA Config: r=8, alpha=16 (merged into base)

DPO Training Log

The model successfully learned to distinguish preferred from rejected outputs, reaching 98.6% accuracy by Step 200.

Performance (Parse Success Rate on public_150 benchmark)

Key Observations

Parse rate vs Evaluation score

While the parse success rate remained at 89.3% (same as SFT), this model achieved the highest evaluation score at the time of training. This indicates that the general DPO improved aspects not captured by parse rate alone:

1. CoT reasoning quality improvement General preference optimization refined the model's Chain-of-Thought reasoning, producing more logical and well-structured analysis before generating output.

2. Output structural accuracy Even when outputs parse successfully, their internal structure (correct field mapping, appropriate nesting, data type handling) improved through preference learning.

3. TOML: 48% → 52% (+4pt) Modest improvement in TOML parse rate. Inline table usage remained high (20/25 tasks), indicating that general DPO alone cannot fully resolve format-specific issues.

Limitation identified

General DPO data does not contain structured-data-specific preferences (e.g. TOML inline table vs section style). This limitation is addressed in DPO Round 2 with format-specific preference pairs.

Usage

Since this is a merged model, you can use it directly with transformers.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Rakushaking/Qwen4b-SFT-d9-merged-after-dpo-d2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = "Convert the following JSON to YAML:\n{\"name\": \"test\", \"value\": 42}"
inputs = tokenizer.apply_chat_template(
    [{"role": "system", "content": "You are an expert in YAML format."},
     {"role": "user", "content": prompt}],
    tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))