qwen2.5-1.5b-rm-ultrafeedback

qwen2.5-1.5b-rm-ultrafeedback

A reward model: Qwen2.5-1.5B-Instruct with a scalar head, trained on 10k UltraFeedback preference pairs (Bradley-Terry) so the preferred response scores higher. LoRA, merged into a standalone model. Part of reward-model-ppo (the classic RLHF loop).

result

Held-out accuracy 0.757 (n=1000): on a held-out pair it scores the preferred response higher 76% of the time. This is a clean, reusable artifact (unlike the PPO policy trained against it, which is a research artifact - see the repo).

use

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("yavuz-ai/qwen2.5-1.5b-rm-ultrafeedback")
rm = AutoModelForSequenceClassification.from_pretrained("yavuz-ai/qwen2.5-1.5b-rm-ultrafeedback", num_labels=1)
msgs = [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
enc = tok.apply_chat_template(msgs, return_tensors="pt", return_dict=True)
reward = rm(**enc).logits[0, 0].item()   # higher = better

training

LoRA (r=16, alpha=32, all-linear) RewardTrainer, 1 epoch over 10k pairs, lr 1e-4, bf16, 1x A100.