Downloads · 30 days
6
6% of all-time downloads
mutsumutsu/dpo-qwen-cot-merged-260208-tokenchg2024-001
dpo-qwen-cot-merged-260208-tokenchg2024-001 is a text generation model from mutsumutsu. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via the Unsloth library.
Downloads · 30 days
6
6% of all-time downloads
All-time downloads
98
Public
Parameters
4B
8.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8 GB · 100%
From the Hugging Face model README
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via the Unsloth library.
This repository contains the full-merged 16-bit weights. No adapter loading is required.
This model has been optimized using DPO to align its responses with preferred outputs, focusing on improving reasoning (Chain-of-Thought) and structured response quality based on the provided preference dataset.
Since this is a merged model, you can use it directly with transformers.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "your_id/your-repo-name"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto"
)
# chane parameter
max_seq_length = 4096
## A100 cnnfig
# 1. Configure DPO Training Arguments
dpo_config = DPOConfig(
learning_rate = 1e-6,
per_device_train_batch_size = 4,
gradient_accumulation_steps = 16,
num_train_epochs = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
warmup_ratio = 0.1,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
output_dir = "dpo_checkpoints_001_A100-2",
beta = 0.1,
max_length = 4096,
max_prompt_length = 2048,
seed = 42,
report_to = "none",
)
# Test inference
prompt = "Your question here"
inputs = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))