Downloads · 30 days
12
19% of all-time downloads
sarimahsan101/Qwen2.5-0.5B-DPO-detox
Qwen2.5-0.5B-DPO-detox is a machine learning model from sarimahsan101. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A DPO (Direct Preference Optimization) fine-tune of Qwen/Qwen2.5-0.5B-Instruct, trained to reduce toxic completions. This is the DPO counterpart to a prior PPO-based detox project (sarimahsan101/qwen2.5-0.5b-detox-ppo…
Downloads · 30 days
12
19% of all-time downloads
All-time downloads
62
Public
Parameters
494M
1000 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors988 MB · 99%
From the Hugging Face model README
A DPO (Direct Preference Optimization) fine-tune of Qwen/Qwen2.5-0.5B-Instruct, trained to reduce toxic completions. This is the DPO counterpart to a prior PPO-based detox project (sarimahsan101/qwen2.5-0.5b-detox-ppo) — same base model, same toxicity classifier, and the same prompt source, so any difference in behavior is attributable to the optimization algorithm rather than a different setup.
This is the full-scale run (QUICK_TEST=False, 4,000 prompts), superseding an earlier 200-prompt smoke test of the same pipeline.
Qwen/Qwen2.5-0.5B-Instructtrl.DPOTrainersigmoid (standard DPO loss)DPOTrainer as a frozen copy of the base checkpoint (no separate ref_model passed)sarimahsan101/Qwen2.5-0.5B-DPO-detoxNo off-the-shelf "detox preference dataset" exists, so one was built from scratch:
allenai/real-toxicity-prompts, sorted by prompt toxicity score and taking the most toxicity-eliciting prompts — 4,000 prompts, toxicity range ≈ 0.892–0.992.do_sample=True, temperature=1.0, top_p=0.9, max_new_tokens=32).facebook/roberta-hate-speech-dynabench-r4-target (P(hate), same classifier used as the PPO run's reward signal).chosen and the most-toxic became rejected; pairs with a toxicity gap smaller than MIN_TOXICITY_GAP = 0.25 were discarded as too weak a signal.This produced 671 preference pairs out of 4,000 prompts (16.8% keep rate), split 90/10 into 603 train / 68 eval examples.
| Hyperparameter | Value |
|---|---|
| Epochs | 3 |
| Per-device train batch size | 4 |
| Gradient accumulation steps | 4 (effective batch size 16) |
| Learning rate | 5e-6 |
beta (KL penalty strength) | 0.1 |
loss_type | sigmoid |
max_length | 128 |
| Precision | bf16=True, fp16=False |
| Seed | 42 |
Total optimizer steps: 114 (3 epochs over 603 examples at effective batch size 16).
train_loss: 0.569
train_runtime: 695.6s (~11.6 min)
epoch: 3.0
Base vs. DPO-tuned completions were compared on the full 68-example eval split, scored with the same toxicity classifier:
| Run | Mean toxicity | Median toxicity | % > 0.5 |
|---|---|---|---|
| Base Qwen2.5-0.5B-Instruct (no tuning) | 0.138 | 0.003 | 11.8% |
| DPO | 0.096 | 0.004 | 7.4% |
Interpretation:
Research artifact for comparing RLHF-style alignment methods (PPO vs. DPO vs. planned IPO/KTO) on a toxicity-reduction task, on free-tier compute. Not intended for production moderation or safety-critical deployment — the toxicity classifier used as a training/eval signal is a single automated proxy, not ground truth.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "sarimahsan101/Qwen2.5-0.5B-DPO-detox"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
inputs = tokenizer("Your prompt here", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=32, do_sample=True, temperature=1.0, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))
facebook/roberta-hate-speech-dynabench-r4-target), which has its own biases and blind spots — it is used here as a proxy reward for both training and evaluation, so the reported improvement is with respect to that specific classifier, not an independent measure of harm.