Downloads · 30 days
5
7% of all-time downloads
Honejudo1234/dpo-qwen-cot-merged_exp09a
dpo-qwen-cot-merged_exp09a is a text generation model from Honejudo1234. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via the Unsloth library.
Downloads · 30 days
5
7% of all-time downloads
All-time downloads
72
Public
Parameters
4B
8.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8 GB · 100%
From the Hugging Face model README
This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 using Direct Preference Optimization (DPO) via the Unsloth library.
This repository contains the full-merged 16-bit weights. No adapter loading is required.
Training Dataset: Honejudo1234/dpo-dataset-qwen-cot-upsampled
This dataset is an upsampled version of the original u-10bei/dpo-dataset-qwen-cot dataset. The upsampling was performed to address class imbalance and improve model performance on underrepresented categories.
Key details:
This model has been optimized using DPO to align its responses with preferred outputs, focusing on improving reasoning (Chain-of-Thought) and structured response quality based on the provided preference dataset.
| Parameter | Value |
|---|---|
| Base Model | Qwen/Qwen3-4B-Instruct-2507 |
| Method | DPO (Direct Preference Optimization) |
| Learning Rate | 5e-07 |
| Beta | 0.2 |
| Epochs | 1 |
| Max Sequence Length | 1024 |
| Parameter | Value |
|---|---|
| Per Device Train Batch Size | 2 |
| Gradient Accumulation Steps | 4 |
| Effective Batch Size | 8 |
Note: The effective batch size is calculated as:
Effective Batch Size = per_device_batch_size × gradient_accumulation_steps × num_gpus
| Parameter | Value |
|---|---|
| LoRA Rank (r) | 8 |
| LoRA Alpha | 16 |
| LoRA Dropout | 0 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Status | Merged into base model (16-bit) |
| Parameter | Value |
|---|---|
| Optimizer | OptimizerNames.ADAMW_8BIT |
| Weight Decay | 0.01 |
| Warmup Ratio | 0.1 |
Since this is a merged model, you can use it directly with transformers.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "your_id/your-repo-name"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto"
)
# Test inference
prompt = "Your question here"
inputs = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))