PairQual-Scorer-zh

PairQual-Scorer-zh

Model summary

This evaluator is designed to assess the overall quality of Chinese corpora. It was trained on 500k annotations generated by GPT-4o-mini, using samples from various sources such as Weibo, Weixin, and Toutiao.

How to use in transformers

To load this model, use the following code:

from transformers import BertConfig, AutoTokenizer
from modeling_bert_regression import BertForRegression

tokenizer = AutoTokenizer.from_pretrained("zks2856/PairQual-Scorer-zh")
config = BertConfig.from_pretrained("zks2856/PairQual-Scorer-zh")
model = BertForRegression.from_pretrained("zks2856/PairQual-Scorer-zh", config=config, trust_remote_code=True)

text = "This is a test sentence."
inputs = tokenizer(text, return_tensors="pt", padding="longest", truncation=True)
pooled, score = model(**inputs)

print(score)

After training with the pairwise paradigm, the scores mostly range from -8 to 8, where a higher score indicates higher quality. Users are encouraged to check the score distribution of their corpora and determine an appropriate threshold based on their own criteria.This model has only been trained to evaluate the quality of Chinese samples. Using it on other languages may lead to undefined behavior.

Training

The evaluator was trained on 500,000 pairs of Chinese samples along with their quality preferences, generated by GPT-4o-mini. The training followed a pairwise learning paradigm that simultaneously increases the score of the higher-quality sample and decreases that of the lower-quality counterpart.

Below is the prompt used for GPT annotations:

<div style="background-color: #fffae6; padding: 10px; border-left: 5px solid orange;">

你是一位专业的中文文本质量评估专家。请综合考虑以下维度对文本的质量进行评估：

文本可读性与内容连贯性：考虑文本的语言是否流畅，是否易于理解，是否格式杂乱或包含乱码，是否使用了恰当的语法和词汇。
教育价值：考虑文本是否包含足够且准确的科学知识和客观事实，对人类具有教育意义，避免逻辑矛盾或虚假信息。
数据清洁度：考虑文本是否已经去除了噪音、主题无关信息、个人敏感信息、广告、链接、版权信息等。

请根据这些评估标准对以下两段文本的质量进行评估。文本可能已经被截断，评估质量时请不要被文本的长度或排列顺序所影响。

文本0:
{{text0}}

文本1:
{{text1}}

请直接输出质量较高的文本序号，不要输出其他内容。

</div>

We added a regression head to bge-base-zh-v1.5 and trained the model for 3 epochs with a learning rate of 1e-5.

Training Details:

• Model: bge-base-zh-v1.5 with a regression head
• Loss: binary cross-entropy loss
• Dataset: 500,000 samples from GPT annotations
• Epochs: 3
• Learning Rate: 1e-5