Downloads · 30 days
0
qwen2.5-1.5b-ultrafeedback-rdpo-adapters
sbc799/qwen2.5-1.5b-ultrafeedback-rdpo-adapters
qwen2.5-1.5b-ultrafeedback-rdpo-adapters is a text generation model from sbc799. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as mit.
This repository contains LoRA adapters and a reward-probe head trained for offline preference optimization on UltraFeedback Binarized.
Access
Public
Updated May 25, 2026
Repo size
126 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors111 MB · 69%
At a glance
- Task
- Text Generation
- Library
- peft
- License
- mit
- Access
- Public
- Created
- May 25, 2026
- Updated
- May 25, 2026
- SHA
- 8256beae
Try a prompt
Base models
Model card
From the Hugging Face model README
Qwen2.5-1.5B UltraFeedback RDPO Adapters
This repository contains LoRA adapters and a reward-probe head trained for offline preference optimization on UltraFeedback Binarized.
Code
Training and evaluation code is available here:
Base Model
- Base model:
Qwen/Qwen2.5-1.5B-Instruct
Included Artifacts
dpo/final: DPO LoRA adaptersimpo/final: SimPO LoRA adapterrdpo/final: RDPO LoRA adapterreward_probe/final: reward-probe head used by RDPOexperiment.yaml: training configuration
Method
RDPO first trains a lightweight reward probe on frozen hidden representations, then distills the probe's chosen-vs-rejected margin into the policy likelihood margin during SimPO-style preference optimization.
Dataset
- Training data:
HuggingFaceH4/ultrafeedback_binarized - Split:
train_prefs - Evaluation split:
test_prefs
License
Please follow the licenses of the base model, dataset, and this repository.
- Task
- Text Generation
- Library
- peft
- License
- mit
- Languages
- en
- Created
- May 25, 2026
- Updated
- May 25, 2026