Downloads · 30 days
4
4% of all-time downloads
FujiwaraAyumu/dpo-qwen-cot-merged
dpo-qwen-cot-merged is a text generation model from FujiwaraAyumu. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via the Unsloth library.
Downloads · 30 days
4
4% of all-time downloads
All-time downloads
97
Public
Repo size
11.4 MB
Likes
0
Public
Click a slice to open those files.
.json14.2 MB · 89%
From the Hugging Face model README
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via the Unsloth library.
This repository contains the full-merged 16-bit weights. No adapter loading is required.
This model has been optimized using DPO to align its responses with preferred outputs, focusing on improving reasoning (Chain-of-Thought) and structured response quality based on the provided preference dataset.
| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen3-4B-Instruct-2507 |
| Method | DPO (Direct Preference Optimization) |
| Epochs | 1 |
| Learning rate | 5e-06 |
| Beta | 0.1 |
| Max sequence length | 1024 |
| LoRA | r=8, alpha=16 (merged into base) |
Since this is a merged model, you can use it directly with transformers.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "FujiwaraAyumu/dpo-qwen-cot-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto",
)
prompt = "Your question here"
inputs = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
).to("cuda")
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))