PairJudge-RM

PairJudge RM

PairJudge RM is a pairwise judge reward model designed to enhance Best-of-N sampling for mathematical reasoning tasks. Instead of assigning arbitrary absolute scores to candidate solutions, PairJudge RM compares them in pairs using chain-of-thought (CoT) reasoning and selects the best answer via a knockout tournament strategy.

• Paper: https://arxiv.org/abs/2501.13007
• Code: https://github.com/THU-KEG/PairJudgeRM
• Dataset: https://huggingface.co/datasets/THU-KEG/PairJudge-432K

Overview

• Pairwise Judgment: Evaluates two candidate solutions simultaneously to determine which is more correct.
• Chain-of-Thought Reasoning: Leverages CoT to transparently verify each step of the candidate solutions.

Model Architecture & Training

PairJudge RM is built by fine-tuning a pre-trained language model (e.g., Qwen-2.5-7B-Instruct) on the PAIRJUDGE-432K dataset. Key training details include:

• Optimizer: Adam
• Learning Rate: 1×10⁻⁵
• Batch Size: 128
• Epochs: 8

Usage

Below is an example of how to use PairJudge RM for evaluating candidate solutions:

from transformers import AutoTokenizer, AutoModelForCausalLM

# template file is avaliable in [https://github.com/THU-KEG/PairwiseRM/blob/main/prompt/compare_0_ex.md]
TEMPLATE = open("prompts/compare_0_ex.md", "r").read()

# Load the tokenizer and model from Hugging Face
tokenizer = AutoTokenizer.from_pretrained("THU-KEG/PairJudgeRM")
model = AutoModelForCausalLM.from_pretrained("THU-KEG/PairJudgeRM")

# Example math problem and candidate solutions
question = "If one equilateral triangle in a regular hexagon has a perimeter of 21 inches, what is the hexagon’s perimeter?"
response_a = "Each side is 7 inches; hexagon perimeter is 42 inches."
response_b = "The triangle's perimeter is 21 inches; hexagon perimeter is 126 inches."

# Construct the input prompt for pairwise judgment
input_text = template.format(question=question, response_a=response_a, response_b=response_b)
inputs = tokenizer(input_text, return_tensors="pt")

# Generate the judgment with a chain-of-thought explanation
outputs = model.generate(**inputs, max_new_tokens=2048)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you find our work useful, please consider citing our paper:

@article{liu2025PairJudge,
  title={PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament},
  author={Liu, Yantao and Yao, Zijun and Min, Rui and Cao, Yixin and Hou, Lei and Li, Juanzi},
  journal={arXiv preprint arXiv:2501.13007},
  year={2025},
  note={in progress work},
  url={https://doi.org/10.48550/arXiv.2501.13007}
}