dpo-qwen-cot-merged

y-tani/dpo-qwen-cot-merged

This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using a two-stage pipeline:

1. SFT (Supervised Fine-Tuning): LoRA adapter from y-tani/lora-qwen3-4b-sft
2. DPO (Direct Preference Optimization): Applied on top of the SFT model

This repository contains the full-merged 16-bit weights. No adapter loading is required.

Training Configuration

• Base model: Qwen/Qwen3-4B-Instruct-2507
• SFT adapter: y-tani/lora-qwen3-4b-sft
• DPO dataset: u-10bei/dpo-dataset-qwen-cot
• Method: SFT → DPO (two-stage)
• Epochs: 1
• Learning rate: 1e-7
• Beta: 0.1
• Max sequence length: 1024
• DPO LoRA Config: r=8, alpha=16 (merged into SFT base)

Sources & License (IMPORTANT)

• Training Data: [u-10bei/dpo-dataset-qwen-cot]
• License: MIT License. (As per dataset terms).
• Compliance: Users must follow the original base model's license terms.