Downloads · 30 days
9
3% of all-time downloads
kaleinaNyan/jina-v3-rullmarena-judge
jina-v3-rullmarena-judge is a machine learning model from kaleinaNyan. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This model is trained to replicate the judgement patterns of GPT-4-1106-Preview in the Russian LLM Arena, designed for faster and more cost-effective evaluation of language models. While the model's focus is on Russia…
Downloads · 30 days
9
3% of all-time downloads
All-time downloads
269
Public
Parameters
614M
1.3 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors1.3 GB · 100%
How the weights are stored.
BF16572M · 93%
From the Hugging Face model README
This model is trained to replicate the judgement patterns of GPT-4-1106-Preview in the Russian LLM Arena, designed for faster and more cost-effective evaluation of language models. While the model's focus is on Russian LLM evaluation, it can also be used for English-centric models.
jina-embeddings-v3 encoder for feature extraction, followed by 4 transformer-decoder blocks.A>>B, A>B, A=B, B>A, B>>A), the model consolidates them into three simplified classes:
The validation process was based on existing judgements from the Russian LLM Arena, which were already available. These judgements were filtered and simplified to match the three-class structure used in training.
Models evaluated:
Validation Performance:
For the test phase, new judgements were generated using GPT-4 for the kolibri-mistral-0427-upd model.
Test Performance:
Upon reviewing erroneous predictions, the following observations were made:
While there is potential to improve alignment between this model and GPT-4, achieving an accuracy beyond 85% is unlikely due to the inherent noise in the benchmarks.
from transformers import AutoModel
jina = AutoModel.from_pretrained("kaleinaNyan/jina-v3-rullmarena-judge", trust_remote_code=True)
prompt_template = """
<user prompt>
{user_prompt}
<end>
<assistant A answer>
{assistant_a}
<end>
<assistant B answer>
{assistant_b}
<end>
""".strip()
prompt = "your prompt"
assistant_a = "assistant a response"
assistant_b = "assistant b response"
example = prompt_template.format(
user_prompt=user_prompt,
assistant_a=assistant_a,
assistant_b=assistant_b,
)
judgement = jina([example])[0].argmax()
judgement_map = {
0: "A is better than B",
1: "A == B",
2: "B is better than A"
}
print(judgement_map[judgement])