Downloads · 30 days
6
4% of all-time downloads
kaleinaNyan/jina-v3-rullmarena-judge-300924
jina-v3-rullmarena-judge-300924 is a machine learning model from kaleinaNyan. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This model is trained to replicate the judgement patterns of GPT-4-1106-Preview in the Russian LLM Arena, designed for faster and more cost-effective evaluation of language models. While the model's focus is on Russia…
Downloads · 30 days
6
4% of all-time downloads
All-time downloads
160
Public
Parameters
589M
1.2 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors1.2 GB · 100%
How the weights are stored.
BF16572M · 97%
From the Hugging Face model README
This model is trained to replicate the judgement patterns of GPT-4-1106-Preview in the Russian LLM Arena, designed for faster and more cost-effective evaluation of language models. While the model's focus is on Russian LLM evaluation, it can also be used for English-centric models.
This is a small upgrade to the kaleinaNyan/jina-v3-rullmarena-judge model:
Surprisingly, these changes gave a tangible performance improvement, so I decided to upload the model. As it turned out (after evaluation on the train set), previous model was not expressive enough.
The validation process was based on existing judgements from the Russian LLM Arena, which were already available. These judgements were filtered and simplified to match the three-class structure used in training.
NOTE: values in parenthesis show relative improvement compared to previous model.
Models evaluated:
Validation Performance:
For the test phase, new judgements were generated using GPT-4 for the kolibri-mistral-0427-upd model.
Test Performance:
from transformers import AutoModel
jina = AutoModel.from_pretrained("kaleinaNyan/jina-v3-rullmarena-judge-300924", trust_remote_code=True)
prompt_template = """
<user prompt>
{user_prompt}
<end>
<assistant A answer>
{assistant_a}
<end>
<assistant B answer>
{assistant_b}
<end>
""".strip()
prompt = "your prompt"
assistant_a = "assistant a response"
assistant_b = "assistant b response"
example = prompt_template.format(
user_prompt=user_prompt,
assistant_a=assistant_a,
assistant_b=assistant_b,
)
judgement = jina([example])[0].argmax()
judgement_map = {
0: "A is better than B",
1: "A == B",
2: "B is better than A"
}
print(judgement_map[judgement])
The ranking was obtained using a modified Russian LLM Arena code. All judgements were regenerated using the jina-judge model.
| Model | Score | 95% CI | Average #Tokens |
|---|---|---|---|
| gpt-4-1106-preview | 81.6 | (-2.3, 3.0) | 541 |
| gpt-4.0-mini | 76.0 | (-2.7, 2.4) | 448 |
| qwen-2.5-72b-it | 72.5 | (-3.6, 3.6) | 557 |
| gemma-2-9b-it-sppo-iter3 | 72.1 | (-3.7, 3.6) | 569 |
| gemma-2-27b-it | 71.1 | (-3.3, 3.2) | 482 |
| gemma-2-9b-it | 70.8 | (-3.4, 3.5) | 569 |
| t-lite-instruct-0.1 | 68.3 | (-3.8, 4.5) | 810 |
| suzume-llama-3-8b-multilingual-orpo | 62.9 | (-3.9, 4.0) | 682 |
| glm-4-9b-chat | 60.5 | (-3.9, 4.0) | 516 |
| sfr-iterative-dpo-llama-3-8b-r | 59.9 | (-4.0, 4.3) | 682 |
| c4ai-command-r-v01 | 56.9 | (-4.2, 3.8) | 516 |
| phi-3-medium-4k-instruct | 56.4 | (-2.8, 3.3) | 566 |
| mistral-nemo-instruct-2407 | 56.1 | (-2.9, 3.4) | 682 |
| yandex_gpt_pro | 51.7 | (-3.4, 3.4) | 345 |
| suzume-llama-3-8b-multilingual | 51.3 | (-3.4, 4.0) | 489 |
| hermes-2-theta-llama-3-8b | 50.9 | (-3.2, 3.4) | 485 |
| starling-1m-7b-beta | 50.2 | (-3.3, 3.4) | 495 |
| gpt-3.5-turbo-0125 | 50.0 | (0.0, 0.0) | 220 |
| llama-3-instruct-8b-sppo-iter3 | 49.8 | (-3.4, 4.0) | 763 |
| llama-3-8b-saiga-suzume-ties | 48.2 | (-4.1, 3.9) | 569 |
| llama-3-smaug-8b | 46.6 | (-3.9, 3.8) | 763 |
| vikhr-it-5.4-fp16-orpo-v2 | 46.6 | (-3.7, 4.0) | 379 |
| aya-23-8b | 46.3 | (-3.8, 3.9) | 571 |
| saiga-llama3-8b_v6 | 45.5 | (-3.8, 3.9) | 471 |
| vikhr-it-5.2-fp16-cp | 43.8 | (-3.9, 4.0) | 543 |
| qwen2-7b-instruct | 43.7 | (-2.5, 2.7) | 492 |
| opencchat-3.5-0106 | 43.4 | (-3.3, 3.7) | 485 |
| gpt-3.5-turbo-1106 | 41.7 | (-2.9, 3.5) | 220 |
| kolibri-mistral-0427-upd | 41.5 | (-3.2, 3.5) | 551 |
| paralex-llama-3-8b-sft | 40.6 | (-3.8, 3.3) | 688 |
| mistral-7b-instruct-v0.3 | 40.3 | (-3.3, 3.4) | 469 |
| llama-3-instruct-8b-simpo | 40.2 | (-2.9, 3.7) | 551 |
| gigachat_pro | 40.2 | (-3.2, 3.5) | 294 |
| hermes-2-pro-llama-3-8b | 39.5 | (-2.9, 3.4) | 689 |
| vikhr-it-5.3-fp16-32k | 39.5 | (-2.8, 3.2) | 519 |
| opencchat-3.6-8b-2204522 | 37.7 | (-3.3, 3.7) | 409 |
| meta-llama-3-8b-instruct | 37.5 | (-3.1, 3.5) | 450 |
| kolibri-vikhr-mistral-0427 | 37.1 | (-3.1, 3.8) | 488 |
| neural-chat-v3.3 | 36.5 | (-2.7, 3.6) | 523 |
| vikhr-it-5.1-fp16 | 36.4 | (-3.5, 3.5) | 448 |
| gigachat-lite | 36.0 | (-2.8, 3.0) | 523 |
| saiga-7b | 25.9 | (-3.1, 3.7) | 927 |
| storm-7b | 25.1 | (-3.6, 4.1) | 419 |
| snorkel-mistral-pairrm-dpo | 16.5 | (-3.8, 3.2) | 773 |