Cold-Start-MetaRM-RM-NLHF-Qwen-32B

💡 Reward Modeling from Natural Language Human Feedback

<p align="left"> <a href="https://arxiv.org/abs/2601.07349"> <img src="https://img.shields.io/badge/arXiv-RM--NLHF-red?logo=arxiv" style="display: inline-block; vertical-align: middle;" alt="RM-NLHF Paper on arXiv" /> </a> <a href="https://github.com/Tongyi-ConvAI/Qwen-Character/tree/main/Character-GenRM-NLHF" target="_blank" style="margin: 2px;"> <img alt="Github" src="https://img.shields.io/badge/Github -RM--NLHF--Codebase-536af5?color=536af5&logo=github" style="display: inline-block; vertical-align: middle;" alt="RM-NLHF Codebase" /> </a> <a href="https://huggingface.co/datasets/Tongyi-ConvAI/RM-NLHF" target="_blank" style="margin: 2px;"> <img alt="HF Model: RM-NLHF" src="https://img.shields.io/badge/%F0%9F%A4%97%20_Dataset-RM--NLHF-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;" alt="HF Model: RM-NLHF" /> </a> </p>

This is the official model repository for the paper "Reward Modeling from Natural Language Human Feedback".

---

🔑 Key Features

• 🥇 First generative reward modeling approach that leverages natural language human critique as training signal.
• 🤝 Hybrid training strategy — jointly utilizes human-written critiques and a specially trained MetaRM for samples without human annotations, achieving SOTA generative reward modeling performance.

---

💾 Checkpoints

We release multiple model variants. All checkpoints are available in our collection:

📦 Collection: Tongyi-ConvAI/rm-nlhf

🟢 Final GRM — Generative Reward Models

Ready-to-use generative reward models trained with the full RM-NLHF pipeline.

🔵 Cold-Start MetaRM

The cold-start MetaRM described in the paper, used as initial weights for MetaRM.

🟡 Final MetaRM

Final-step MetaRM checkpoints co-trained alongside the generative reward model.

⚪ Baseline — Outcome Reward Model

A baseline trained solely on outcome labels, without natural language critique.

---

🧷 Citation

@misc{wang2026rewardmodelingnaturallanguage,
  title={Reward Modeling from Natural Language Human Feedback}, 
  author={Zongqi Wang and Rui Wang and Yuchuan Wu and Yiyao Yu and Pinyi Zhang and Shaoning Sun and Yujiu Yang and Yongbin Li},
  year={2026},
  eprint={2601.07349},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2601.07349}, 
}