Llama-3_3-Nemotron-Super-49B-GenRM

Model Overview

Description:

Llama-3.3-Nemotron-Super-49B-GenRM is a generative reward model that leverages Llama-3.3-Nemotron-Super-49B-v1 as the foundation and is fine-tuned using Reinforcement Learning to predict the quality of LLM generated responses.

Llama-3.3-Nemotron-Super-49B-GenRM can be used to judge the quality of one response, or the ranking between two responses given an English conversation history. It will first generate reasoning traces then output an integer score. A higher score means the response is of higher quality.

See details on how this model was trained at https://arxiv.org/abs/2505.11475

This model is ready for commercial/non-commercial use.

License/Terms of Use:

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License . Additional Information: Llama 3.3 Community License Agreement. Built with Llama.

Deployment Geography

Global

Use Case:

Llama-3.3-Nemotron-Super-49B-GenRM can be used to judge the quality of one response, or the ranking between two responses given an English conversation history. It will first generate reasoning traces then output an integer score.

Release Date:

HuggingFace 06/27/2025 via https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-GenRM

References:

• HelpSteer3-Preference
• HelpSteer2-Preference
• SteerLM method
• HelpSteer
• HelpSteer2
• Llama-Nemotron: Efficient Reasoning Models
• The future of AI: Built with Llama
• Meta's Llama 3.3 Webpage
• Meta's Llama 3.3 Model Card

RM-Bench LeaderBoard

As of 15 May 2025, our reward models trained with HelpSteer3-Preference are the top performing Bradley-Terry reward models on RM-Bench, an improved variant of RewardBench for evaluating Reward Models in Chat, Math, Code and Safety. Our GenRMs also outperform the corresponding Bradley-Terry reward models.

Note that Skywork-Reward-Llama-3.1-8B was the best performing reward model reported on RM-Bench and we evaluated all other models.

JudgeBench LeaderBoard

As of 15 May 2025, our reward models trained with HelpSteer3-Preference are the top performing Bradley-Terry reward models on JudgeBench, a popular benchmark for evaluating LLM-as-a-judge applications relating to General Knowledge, Logical Reasoning, Math and Coding. Our GenRMs also outperform the corresponding Bradley-Terry reward models.

Note that Skywork-Reward-Gemma-2-27B was the best performing reward model reported on JudgeBench and we evaluated all other numbers.

Model Architecture:

Architecture Type: Transformer <br> Network Architecture: Llama-3.3-Nemotron-Super-49B-v1 <br>

We developed this model using Llama-3.3-Nemotron-Super-49B-v1 as its foundation. This model contains 49 billion parameters.

Input:

Input Type(s): Text <br> Input Format: String <br> Input Parameters: One Dimensional (1D) <br> Other Properties Related to Input: Max of 128k tokens<br>

Output:

Output Type(s): Text <br> Output Format: String <br> Output Parameters: One-Dimensional (1D) <br> Other Properties Related to Output: The output contains a reasoning trace and a final score. <br>

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>

Software Integration:

Runtime Engine(s): <br>

• vLLM 0.8.3 <br>

Supported Hardware Microarchitecture Compatibility: <br>

• NVIDIA Ampere <br>
• NVIDIA Hopper <br>

Supported Operating System(s): Linux <br>

Quick Start

We recommend serving the model with vLLM. You can use the model with 2 or more 80GB GPUs (NVIDIA Ampere or newer) with at least 100GB of free disk space to accommodate the download.

pip install vllm==0.8.3

python3 -m vllm.entrypoints.openai.api_server \
  --model "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM" \
  --trust-remote-code \
  --seed=1 \
  --host="0.0.0.0" \
  --port=5000 \
  --served-model-name "nvidia/Llama-3_3-Nemotron-Super-49B-GenRM" \
  --tensor-parallel-size=8 \
  --max-model-len=40000 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager

Now you can query the model, here is an example:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:5000/v1", api_key="dummy")

# when judging one response
msg = [
  {"role": "user", "content": "What is 1+1?"}, 
  {"role": "assistant", "content": "1+1=2"}, 
  {"role": "user", "content": "What about 1+2?"},
  {"role": "response_1", "content": "1+2=4"}
]

completion = client.chat.completions.create(
    model="nvidia/Llama-3_3-Nemotron-Super-49B-GenRM",
    messages=msg,
    temperature=0.0,
    top_p=1.0,
    max_tokens=32768,
    stream=False
)
output = completion.choices[0].message.content
print(output.split("</think>")[-1].strip())
"""
[The Begin of Analysis on Response 1]
Response 1 claims that 1+2 equals 4, which is mathematically incorrect (the correct answer is 3). While the response is clear and formatted appropriately, its inaccuracy makes it mostly unhelpful. However, it attempts to answer the question in the expected format, so it is borderline unhelpful rather than completely unhelpful.
[The End of Analysis on Response 1]

[The Begin of Individual Scores]
\boxed{2}
[The End of Individual Scores]
"""
# when judging two responses
msg = [
  {"role": "user", "content": "What is 1+1?"}, 
  {"role": "assistant", "content": "1+1=2"}, 
  {"role": "user", "content": "What about 1+2?"},
  {"role": "response_1", "content": "1+2=4"},
  {"role": "response_2", "content": "1+2=3"}
]

completion = client.chat.completions.create(
    model="nvidia/Llama-3_3-Nemotron-Super-49B-GenRM",
    messages=msg,
    temperature=0.0,
    top_p=1.0,
    max_tokens=32768,
    stream=False
)
output = completion.choices[0].message.content
print(output.split("</think>")[-1].strip())
"""
[The Begin of Analysis on Response 1]
Response 1 claims "1+2=4", which is mathematically incorrect. While it is clear and relevant to the topic, it fails in correctness, instruction following, and providing a useful answer. It is mostly unhelpful except for being on-topic.
[The End of Analysis on Response 1]

[The Begin of Analysis on Response 2]
Response 2 correctly states "1+2=3". It is accurate, clear, relevant, and fully fulfills the user’s request. While it lacks extra detail, the simplicity is appropriate for the question, making it mostly helpful.
[The End of Analysis on Response 2]

[The Begin of Individual Scores]
\boxed{2, 4}
[The End of Individual Scores]

[The Begin of Ranking Score]
\boxed{6}
[The End of Ranking Score]
"""

Note that the conversation history should be presented in "user" and "assistant" roles, where the last turn is user turn. The responses to be judged should be in "response_1" (and "response_2") roles.

Interpretation of Scores

When judging one response, the model will generate a helpfulness score from 1 to 5, where higher is better.

When judging two responses, the model will generate an individual helpfulness score for each response, then a ranking score. The ranking score is a number between 1 and 6, where:

1 = Response 1 is much better than Response 2

2 = Response 1 is better than Response 2

3 = Response 1 is slightly better than Response 2

4 = Response 2 is slightly better than Response 1

5 = Response 2 is better than Response 1

6 = Response 2 is much better than Response 1

For details, please see Appendix J in our paper.

Model Version:

v1.0

Training, Testing and Evaluation Datasets:

Training Datasets:

Dataset Name: HelpSteer3 <br> Dataset Link: https://huggingface.co/datasets/nvidia/HelpSteer3

Data Collection Method by dataset <br>

• [Hybrid: Human, Synthetic] <br>

Labeling Method by dataset <br>

• [Human] <br>

Properties: <br>

• 38,459 prompts, each with a pair of responses as well as human preferences between the pair of responses.

Testing Datasets:

Dataset Name: HelpSteer3 <br> Dataset Link: https://huggingface.co/datasets/nvidia/HelpSteer3

Data Collection Method by dataset <br>

• [Hybrid: Human, Synthetic] <br>

Labeling Method by dataset <br>

• [Human] <br>

Properties: <br>

• 2,017 prompts, each with a pair of responses as well as human preferences between the pair of responses.

Evaluation Datasets

Dataset Name: RM-Bench <br> Dataset Link: https://huggingface.co/datasets/THU-KEG/RM-Bench

Data Collection Method by dataset <br>

• [Hybrid: Human, Synthetic] <br>

Labeling Method by dataset <br>

• [Hybrid: Human, Synthetic] <br>

Properties: <br>

• 1,327 prompts, each with three pairs of responses as well as preferences between the pair of responses.

Dataset Name: JudgeBench <br> Dataset Link: https://huggingface.co/datasets/ScalerLab/JudgeBench

Data Collection Method by dataset <br>

• [Hybrid: Human, Synthetic] <br>

Labeling Method by dataset <br>

• [Hybrid: Human, Synthetic] <br>

Properties: <br>

• 350 prompts, each with a pair of responses as well as preferences between the pair of responses.

Inference:

Engine: vLLM <br> Test Hardware: H100, A100 80GB, A100 40GB <br>

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Citation

If you find this model useful, please cite the following work:

@misc{wang2025helpsteer3preferenceopenhumanannotatedpreference,
      title={Help{S}teer3-{P}reference: Open Human-Annotated Preference Data across Diverse Tasks and Languages},
      author={Zhilin Wang and Jiaqi Zeng and Olivier Delalleau and Hoo-Chang Shin and Felipe Soares and Alexander Bukharin and Ellie Evans and Yi Dong and Oleksii Kuchaiev},
      year={2025},
      eprint={2505.11475},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.11475}, 
}