Downloads · 30 days
24
4% of all-time downloads
sarimahsan101/qwen2.5-0.5b-detox-ppo
qwen2.5-0.5b-detox-ppo is a text generation model from sarimahsan101. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
A reinforcement learning (PPO) fine-tuned version of Qwen2.5-0.5B designed to reduce toxic text generation while preserving general language generation capability.
Downloads · 30 days
24
4% of all-time downloads
All-time downloads
605
Public
Parameters
494M
1000 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors988 MB · 99%
How the weights are stored.
BF16494M · 100%
From the Hugging Face model README
A reinforcement learning (PPO) fine-tuned version of Qwen2.5-0.5B designed to reduce toxic text generation while preserving general language generation capability.
The model was trained using TRL's PPOTrainer, with Detoxify serving as the reward model. Prompts were sampled from the RealToxicityPrompts dataset, and the model was rewarded for producing continuations with lower predicted toxicity.
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-0.5B |
| Fine-tuning Method | Proximal Policy Optimization (PPO) |
| Library | TRL 0.11.4 |
| Reward Model | Detoxify (Original) |
| Dataset | allenai/real-toxicity-prompts |
| Prompt Filtering | Toxicity ≥ 0.30 |
| PPO Updates | 156 |
| Batch Size | 128 |
| Mini Batch Size | 16 |
| Reward Function | reward = 1 - toxicity_probability (scaled to [-1, 1]) |
The objective is to minimize generated toxicity through reinforcement learning. For each sampled prompt:
reward = 1 - toxicity.[-1, 1].No supervised fine-tuning (SFT) was performed after PPO.
Evaluation was performed on a held-out subset of 200 prompts from the RealToxicityPrompts dataset. Generated continuations were scored using Detoxify.
| Metric | Base Qwen2.5-0.5B | PPO Fine-tuned |
|---|---|---|
| Mean Toxicity | 0.2631 | 0.2251 |
| Median Toxicity | 0.0570 | 0.0378 |
| Standard Deviation | 0.3474 | 0.3283 |
| Outcome | Percentage |
|---|---|
| Lower toxicity after PPO | 64% |
| Higher toxicity after PPO | 36% |
| No change | 0% |
These results indicate that PPO successfully shifted the policy toward lower-toxicity generations while maintaining coherent text generation.
Training reward increased steadily during optimization.
<p align="center"> <img src="./assets/ppo_reward_curve.png" width="700"> </p>Prompt:
Some people from that country are...
Base Model
... (example output)
Toxicity: 0.xx
PPO Model
... (example output)
Toxicity: 0.xx
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "The person said"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=64,
do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
If you use this model, please cite:
@misc{realtoxicityprompts,
title={RealToxicityPrompts},
author={Gehman et al.},
year={2020}
}