GenRubric-8B

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

<p align="center"> <a href="https://github.com/foggpoy/GenRubric"> <img src="https://img.shields.io/badge/Code-GitHub-black?logo=github" alt="Code"> </a> <a href="https://arxiv.org/abs/2608.29856"> <img src="https://img.shields.io/badge/Paper-arXiv-b31b1b" alt="Paper"> </a> </p>

Introduction

GenRubric is a family of models for query-specific rubric generation, designed to support scalable and fine-grained evaluation of large language models on open-ended tasks. Instead of relying on manually constructed rubrics or additional human annotations, GenRubric improves rubric generation through self-evolution. Experiments across multiple domains show that GenRubric generates rubrics whose induced evaluations better align with expert-written rubrics, while also generalizing well to domains not seen during training.

Models

We release GenRubric at three model scales:

All models are built upon the corresponding Qwen3 Base models.

Usage

GenRubric follows the same chat format as Qwen3 and can be directly loaded with Hugging Face Transformers.

Inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "chenyifan0929/GenRubric-8B"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)

query = """
A company terminates an employee after discovering that the employee
had provided false information during recruitment. The employee argues
that the false information was unrelated to job performance and that
the termination was unlawful. Analyze whether the termination is legally justified.
"""

messages = [
    {
        "role": "user",
        "content": f"Generate a comprehensive evaluation rubric for the following query:\n\n{query}",
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
)

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)

print(response)

The generated output contains query-specific evaluation criteria that can be used to evaluate model responses to the given query.

Citation

If you find GenRubric useful in your research, please consider citing our paper:

@misc{chen2026genrubricselfevolvingrubricgeneration,
      title={GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation},
      author={Yifan Chen and Haitao Li and Qingyao Ai and Fengbin Zhu and Tat-Seng Chua and Min Zhang and Yiqun Liu},
      year={2026},
      eprint={2608.29856},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.29856},
}