Rebuttal-RM

Rebuttal-RM 🏅

1. Introduction

Rebuttal-RM is a scoring model trained to automatically assess author responses in light of the target comment and its supporting context, with the explicit goal of matching human‐reviewer preferences. The reward model, denoted GRM, receives as input the retrieved evidence chunks CE, the current review R_i, the target comment target, and a candidate reply response; it returns a vector of rubric-aligned scores s together with an explanatory rationale e. Formally,

$$ (s,e);=;\mathrm{GRM}!\bigl(,\sum_{p_j\in\text{CE}} p_j,; R_i,; c_{\text{target}},; r_{\text{response}}\bigr). $$

To obtain a robust evaluator we curate 102 K training instances drawn from three sources: (i) 12 000 original author rebuttals that serve as a human baseline, (ii) GPT-4.1–refined answers representing an upper quality bound, and (iii) diverse model-generated replies (e.g., Qwen-2.5-3B, Claude-3.5) to broaden stylistic coverage. Using Qwen-3-8B as the backbone, we fine-tune on this corpus to yield the final Rebuttal-RM. As reported in Table 1, Rebuttal-RM achieves the strongest agreement with expert annotators, posting an average correlation of 0.812 and outperforming GPT-4.1 and DeepSeek-R1 by 9.0 % and 15.2 %, respectively.

2. Performance (agreement with human ratings)(higher = better)

3. Deployment / Usage

3.1 Run with vLLM (OpenAI protocol)

pip install "vllm>=0.4.2"
python -m vllm.entrypoints.openai.api_server \
  --model Zhitao-He/Rebuttal-RM \
  --dtype auto \
  --port 8000

3.2 Query via the official openai SDK

python

import openai
openai.api_key  = "EMPTY"
openai.base_url = "http://localhost:8000/v1"

prompt = f"""
The whole review content is{Full_Review_Content}
The target comment content is{Target_Comment}.
The best relevant paper fragment is{Relevant_Paper_Fragment}.
The response is{response}
"""

resp = openai.chat.completions.create(
    model="Zhitao-He/Rebuttal-RM",
    messages=[
        {"role": "system", "content": system_prompt}, 
        {"role": "user",   "content": user_prompt}
    ],
    temperature=0.7
)
print(resp.choices[0].message.content）

3.3 Default System Prompt (system_prompt)

You are a seasoned academic reviewer and response optimization expert. Your task is to evaluate the quality of the response based on the review comments, paper fragments, and the authors' responses. Please strictly follow the requirements below, and output only the score and score explanation.

Input variables:

review_content: Complete content of the review comments. similar_paper_fragments: Best paper fragment most relevant to the comment. comment: Specific segment of the review comments. original_response: The authors' original response text to the comment.

Your task: Based on the input information, output only a JSON object containing the following two items:
Scoring Standard
Score Range: 0 - 10
0: Wholly Ineffective
1-2: Perfunctory
3-4: Unconvincing
5-6: Addresses Some Concerns
7-8: Exceptional
9-10: Outstanding
score: Four-dimensional score breakdown, ranging from 0-10, structured as follows:
Attitude: The tone and professionalism of the response. 
Clarity: The logic, structure, and focus of the response.
Persuasiveness: The effectiveness of the argumentation and evidence support. 
Constructiveness: The commitment to revisions and specific actions taken.

score_explanation: A brief explanation of each score, specifically citing key points from the response text that reflect the scores and any shortcomings.

Output requirements:

Only output the JSON object; do not include any other characters or explanations.
The scoring must be reasonable, and the score explanation must clearly reference the original text that reflects the score. 
All output must be in formal, polite academic English.
Your output must be strictly JSON formal which can be dirctly loaded by json.load()
Output format example:
{ "score": { "Attitude": <int>,
              "Clarity": <int>, 
              "Persuasiveness": <int>,
              "Constructiveness": <int> }, 
  "score_explanation": <explanation for your given score>}"""

4. Citation

@misc{he2026dancingchainsstrategicpersuasion,
      title={Dancing in Chains: Strategic Persuasion in Academic Rebuttal via Theory of Mind}, 
      author={Zhitao He and Zongwei Lyu and Yi R Fung},
      year={2026},
      eprint={2601.15715},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.15715}, 
}