Downloads · 30 days
0
RSPO_Qwen2.5-1.5B
HoangTran223/RSPO_Qwen2.5-1.5B
RSPO_Qwen2.5-1.5B is a machine learning model from HoangTran223. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
Regularized Self-Play Preference Optimization (reverse-KL) on Qwen2.5-1.5B, after SFT on UltraChat200k.
Access
Public
Updated Sep 9, 2026
Repo size
13.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors12.3 GB · 91%
At a glance
- Library
- transformers
- License
- apache-2.0
- Access
- Public
- Created
- Aug 28, 2026
- Updated
- Sep 9, 2026
- SHA
- 06a0bb61
Base models
Model card
From the Hugging Face model README
RSPO Qwen2.5-1.5B (UltraChat50k)
Regularized Self-Play Preference Optimization (reverse-KL) on Qwen2.5-1.5B, after SFT on UltraChat200k.
Round 1 reuses the SPPO iter1 preference parquet. Frozen reference = UltraChat200k SFT.
Final policy: iter3/. Full run is iter1–iter3 (complete).
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("HoangTran223/RSPO_Qwen2.5-1.5B", subfolder="iter3")
| Path | Contents |
|---|---|
iterN/ | Final HF weights after 2 training epochs |
iterN/synthetic_data/ | Preference parquet used by the trainer |
iterN/generated/ | Combined generations + PairRM probabilities (when present) |
iterN/ranking/ | PairRM rank matrix (0_0.npy) |
iterN/logs/ | vLLM generate, PairRM rank, trainer stdout |
configs/ | Trainer yaml |
Training: loss_type=sppo_reversekl, reg_coef=0.5, max_length=1024, max_prompt_length=512, per-device batch 2 × grad accum 2, RMSProp, LR 5e-7, 2 epochs. Frozen SFT ref + last-iterate log-probs precomputed for iter3 (same numerical values as live forwards).
- Library
- transformers
- License
- apache-2.0
- Created
- Aug 28, 2026
- Updated
- Sep 9, 2026