Qwen2.5-7B-MT-GRRM-Optimized

Model Card for double7/Qwen2.5-7B-MT-GRRM-Optimized

Model Details

Model Description

double7/Qwen2.5-7B-MT-GRRM-Optimized is a multilingual Machine Translation (MT) model post-trained with Group Relative Policy Optimization (GRPO) using GRRM (Group Relative Reward Model) as the reward provider. The training goal is to improve translation quality—especially on challenging, reasoning-intensive translation cases—by leveraging groupwise, relative reward signals (GQM) that provide fine-grained intra-group ranking feedback.

The model is initialized from Qwen2.5-7B, then:

1. Cold-started via SFT on Chinese–English data (with LLM-annotated comparative reasoning / CoT-style supervision for translation with reasoning).
2. Optimized via GRPO on multilingual MT data (TowerBlocks, ~150k samples spanning 10 languages) using GRRM as the reference-free reward model, with Cross-Lingual Augmentation (CLA) enabled.

• Model type: Causal Language Model (Instruction-tuned / MT-oriented post-training)
• Primary use: Machine Translation (multilingual, En↔X / Zh↔En emphasized)
• Language(s): English, Portuguese, Spanish, French, German, Dutch, Italian, Russian, Chinese (and potentially other languages, but not guaranteed)
• License: Apache License 2.0
• Finetuned from model: Qwen2.5-7B

Model Sources

• Paper: GRRM: Group Relative Reward Modeling for Machine Translation
• Repository: https://github.com/NJUNLP/GRRM
• Reward model (used in training): Qwen2.5-7B-GRRM

Uses

Direct Use

This model is intended for translation-with-reasoning, including:

• General-domain MT across multiple language pairs (e.g., En↔De/Fr/Es/Pt/It/Nl/Ru/Zh).
• Challenging MT scenarios where reasoning about ambiguity, localization, idioms, discourse coherence, or subtle adequacy issues is required.

Input / Output Format

Input format

Format the input as an instruction-style MT prompt with explicit reasoning request (See the example below). Wrap the source text in a fenced code block to avoid formatting ambiguity and to support multi-line inputs.

Output format

The model typically outputs:

1. A step-by-step analysis section in Markdown, explaining idioms, ambiguity resolution, and translation choices.
2. The final translation wrapped in a fenced code block, as requested by the prompt.

Bias, Risks, and Limitations

• Language and domain coverage: Although trained with multilingual GRPO, quality can degrade for low-resource languages, specialized terminology, or very creative text.
• Prompt sensitivity: Output style may vary with prompting. For consistent outputs, standardize prompts.
• Reward-model-induced bias: Optimization uses GRRM (a learned judge). While GRRM improves ranking sensitivity in groups, it may still encode preferences (e.g., fluency/style) and occasionally mis-rank edge cases, which can influence the final MT model.

How to Get Started with the Model

[!NOTE] For high-throughput inference with vllm and official parsing script, please refer to our code repository.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_PATH = "double7/Qwen2.5-7B-MT-GRRM-Optimized"

tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH, device_map="auto", torch_dtype="auto"
)

prompt = """Translate the following text from English into Chinese. Perform a step by step analysis and output the final translation in a code block.

Source text:
```
The grass is always greener on the other side.
```
"""

messages = [{"role": "user", "content": prompt}]
input_text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=8192,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

<details><summary>Output Example</summary>

### Step-by-step Analysis

1.  **"The grass is always greener on the other side."**: This is a well-known English idiom. A literal, word-for-word translation would be unnatural in Chinese. The core meaning is that people often feel their current situation is not as good as they perceive another person's or another situation to be.

2.  **Identifying the Chinese equivalent**: The most common and idiomatic translation for this concept in Chinese is "这山望着那山高" (zhè shān wàngzhe nà shān gāo), which literally means "To look at the other mountain, thinking it's higher." This perfectly captures the essence of wanting what one doesn't have.

3.  **Exploring other options**:
    *   "彼岸的草总是更绿" (Bǐ'àn de cǎo zǒngshì gèng lǜ): A more literal, poetic translation, "The grass on the other shore is always greener," which is also understandable but not as common in everyday speech as the idiom.
    *   "隔岸观火" (gé àn guān huǒ): This idiom means "to watch a fire from the other bank," implying a passive, detached, or envious observation, but it doesn't fully capture the comparison between one's own situation and another's.

4.  **Final Selection**: "这山望着那山高" is the most idiomatic and widely used equivalent. A slightly more formal and written version is "人往高处走，水往低处流" (rén wǎng gāochù zǒu, shuǐ wǎng dīchù liú), meaning "People strive for higher ground, water flows to lower ground," which also conveys a similar message. However, "这山望着那山高" is the most direct and natural translation.

### Final Translation

```
这山望着那山高。
```

</details>

Training Details

Training Data

• SFT cold-start (Zh–En): Chinese–English subset of TowerBlocks, supervised with LLM-annotated reasoning and translation signals.
• GRPO stage (multilingual): TowerBlocks multilingual translation data covering 10 languages, about 150k training samples.

Training Procedure (High-level)

1. SFT (cold start): Initialize translation and basic reasoning behaviors (Zh–En).
2. GRPO w/ GRRM feedback: Sample groups of candidate translations per source, score/rank them with GRRM (groupwise), compute advantages within-group, and update the policy to prefer better candidates—targeting improved reasoning ability and robustness on challenging MT cases.

Training Hyperparameters

• Hardware: 16 × NVIDIA A100 (80GB)

SFT (policy cold-start)

• Epochs: 2
• Global batch size: 64
• LR scheduler: cosine
• Peak learning rate: 1e-5
• Warmup ratio: 0.1

Reinforcement Learning (policy optimization with GRRM)

• RL algorithm: GSPO (Group Sequence Policy Optimization), with additional stabilization enhancements (see paper appendix)
• Epochs: 1
• Learning rate: 1e-5
• LR scheduler: constant
• Rollouts per prompt: 4
• Length control: max 4096 tokens, soft length penalty with overlong buffer = 2048 tokens
• Total batch size: 512
• PPO mini-batch size: 128
• KL penalty: disabled (no KL divergence penalty)
• Advantage normalization: standard deviation normalization removed
• Reward scaling: scaled to [0, 0.1] for stability

Evaluation

MT performance on WMT and Seed-X-Challenge benchmarks. We report BLEURT-20 and LLM-as-a-Judge scores (evaluated by DeepSeek-R1-0528). Optimizing with GRRM via GRPO significantly improves the translation quality and reasoning capabilities of the base model.

Citation

@article{yang2026grrmgrouprelativereward,
      title={GRRM: Group Relative Reward Modeling for Machine Translation}, 
      author={Sen Yang and Shanbo Cheng and Lu Xu and Jianbing Zhang and Shujian Huang},
      year={2026},
      eprint={2602.14028},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.14028},
}