Downloads · 30 days
259
7% of all-time downloads
OpenLLM-Ro_-_RoMistral-7b-Instruct-2024-05-17-gguf
RichardErkhov/OpenLLM-Ro_-_RoMistral-7b-Instruct-2024-05-17-gguf
OpenLLM-Ro_-_RoMistral-7b-Instruct-2024-05-17-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
All-time downloads
3.7K
Public
Repo size
82.2 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.gguf95.2 GB · 100%
At a glance
- Access
- Public
- Created
- Oct 11, 2024
- Updated
- Oct 11, 2024
- SHA
- 505eaa43
Model card
From the Hugging Face model README
Quantization made by Richard Erkhov.
RoMistral-7b-Instruct-2024-05-17 - GGUF
- Model creator: https://huggingface.co/OpenLLM-Ro/
- Original model: https://huggingface.co/OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17/
Original model description:
license: cc-by-nc-4.0 language:
- ro base_model:
- mistralai/Mistral-7B-v0.1 datasets:
- OpenLLM-Ro/ro_sft_alpaca
- OpenLLM-Ro/ro_sft_alpaca_gpt4
- OpenLLM-Ro/ro_sft_dolly
- OpenLLM-Ro/ro_sft_selfinstruct_gpt4
- OpenLLM-Ro/ro_sft_norobots
- OpenLLM-Ro/ro_sft_orca
- OpenLLM-Ro/ro_sft_camel
model-index:
- name: OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17
results:
- task:
type: text-generation
dataset:
name: RoMT-Bench
type: RoMT-Bench
metrics:
- name: Score type: Score value: 4.99
- task:
type: text-generation
dataset:
name: RoCulturaBench
type: RoCulturaBench
metrics:
- name: Score type: Score value: 3.38
- task:
type: text-generation
dataset:
name: Romanian_Academic_Benchmarks
type: Romanian_Academic_Benchmarks
metrics:
- name: Average accuracy type: accuracy value: 52.54
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_arc_challenge
type: OpenLLM-Ro/ro_arc_challenge
metrics:
- name: Average accuracy type: accuracy value: 50.41
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_mmlu
type: OpenLLM-Ro/ro_mmlu
metrics:
- name: Average accuracy type: accuracy value: 51.61
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_winogrande
type: OpenLLM-Ro/ro_winogrande
metrics:
- name: Average accuracy type: accuracy value: 66.48
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_hellaswag
type: OpenLLM-Ro/ro_hellaswag
metrics:
- name: Average accuracy type: accuracy value: 60.27
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_gsm8k
type: OpenLLM-Ro/ro_gsm8k
metrics:
- name: Average accuracy type: accuracy value: 34.19
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_truthfulqa
type: OpenLLM-Ro/ro_truthfulqa
metrics:
- name: Average accuracy type: accuracy value: 52.30
- task:
type: text-generation
dataset:
name: LaRoSeDa_binary
type: LaRoSeDa_binary
metrics:
- name: Average macro-f1 type: macro-f1 value: 97.36
- task:
type: text-generation
dataset:
name: LaRoSeDa_multiclass
type: LaRoSeDa_multiclass
metrics:
- name: Average macro-f1 type: macro-f1 value: 67.55
- task:
type: text-generation
dataset:
name: LaRoSeDa_binary_finetuned
type: LaRoSeDa_binary_finetuned
metrics:
- name: Average macro-f1 type: macro-f1 value: 98.80
- task:
type: text-generation
dataset:
name: LaRoSeDa_multiclass_finetuned
type: LaRoSeDa_multiclass_finetuned
metrics:
- name: Average macro-f1 type: macro-f1 value: 88.28
- task:
type: text-generation
dataset:
name: WMT_EN-RO
type: WMT_EN-RO
metrics:
- name: Average bleu type: bleu value: 27.93
- task:
type: text-generation
dataset:
name: WMT_RO-EN
type: WMT_RO-EN
metrics:
- name: Average bleu type: bleu value: 13.21
- task:
type: text-generation
dataset:
name: WMT_EN-RO_finetuned
type: WMT_EN-RO_finetuned
metrics:
- name: Average bleu type: bleu value: 28.72
- task:
type: text-generation
dataset:
name: WMT_RO-EN_finetuned
type: WMT_RO-EN_finetuned
metrics:
- name: Average bleu type: bleu value: 40.86
- task:
type: text-generation
dataset:
name: XQuAD
type: XQuAD
metrics:
- name: Average exact_match type: exact_match value: 43.66
- task:
type: text-generation
dataset:
name: XQuAD
type: XQuAD
metrics:
- name: Average f1 type: f1 value: 63.70
- task:
type: text-generation
dataset:
name: XQuAD_finetuned
type: XQuAD_finetuned
metrics:
- name: Average exact_match type: exact_match value: 55.04
- task:
type: text-generation
dataset:
name: XQuAD_finetuned
type: XQuAD_finetuned
metrics:
- name: Average f1 type: f1 value: 72.31
- task:
type: text-generation
dataset:
name: STS
type: STS
metrics:
- name: Average spearman type: spearman value: 77.43
- task:
type: text-generation
dataset:
name: STS
type: STS
metrics:
- name: Average pearson type: pearson value: 78.43
- task:
type: text-generation
dataset:
name: STS_finetuned
type: STS_finetuned
metrics:
- name: Average spearman type: spearman value: 87.25
- task:
type: text-generation
dataset:
name: STS_finetuned
type: STS_finetuned
metrics:
- name: Average pearson type: pearson value: 87.79
- task:
type: text-generation
dataset:
name: RoMT-Bench
type: RoMT-Bench
metrics:
- name: First turn type: Score value: 5.46
- name: Second turn type: Score value: 4.53
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_arc_challenge
type: OpenLLM-Ro/ro_arc_challenge
metrics:
- name: 0-shot type: accuracy value: 47.47
- name: 1-shot type: accuracy value: 48.59
- name: 3-shot type: accuracy value: 50.30
- name: 5-shot type: accuracy value: 51.33
- name: 10-shot type: accuracy value: 52.36
- name: 25-shot type: accuracy value: 52.44
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_mmlu
type: OpenLLM-Ro/ro_mmlu
metrics:
- name: 0-shot type: accuracy value: 50.01
- name: 1-shot type: accuracy value: 50.18
- name: 3-shot type: accuracy value: 53.13
- name: 5-shot type: accuracy value: 53.12
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_winogrande
type: OpenLLM-Ro/ro_winogrande
metrics:
- name: 0-shot type: accuracy value: 64.96
- name: 1-shot type: accuracy value: 67.09
- name: 3-shot type: accuracy value: 67.01
- name: 5-shot type: accuracy value: 66.85
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_hellaswag
type: OpenLLM-Ro/ro_hellaswag
metrics:
- name: 0-shot type: accuracy value: 59.99
- name: 1-shot type: accuracy value: 59.48
- name: 3-shot type: accuracy value: 60.14
- name: 5-shot type: accuracy value: 60.61
- name: 10-shot type: accuracy value: 61.12
- task:
type: text-generation
dataset:
name: OpenLLM-Ro/ro_gsm8k
type: OpenLLM-Ro/ro_gsm8k
metrics:
- name: 1-shot type: accuracy value: 21.68
- name: 3-shot type: accuracy value: 38.21
- name: 5-shot type: accuracy value: 42.68
- task:
type: text-generation
dataset:
name: LaRoSeDa_binary
type: LaRoSeDa_binary
metrics:
- name: 0-shot type: macro-f1 value: 97.27
- name: 1-shot type: macro-f1 value: 96.37
- name: 3-shot type: macro-f1 value: 97.97
- name: 5-shot type: macro-f1 value: 97.83
- task:
type: text-generation
dataset:
name: LaRoSeDa_multiclass
type: LaRoSeDa_multiclass
metrics:
- name: 0-shot type: macro-f1 value: 63.95
- name: 1-shot type: macro-f1 value: 66.89
- name: 3-shot type: macro-f1 value: 68.16
- name: 5-shot type: macro-f1 value: 71.19
- task:
type: text-generation
dataset:
name: WMT_EN-RO
type: WMT_EN-RO
metrics:
- name: 0-shot type: bleu value: 24.87
- name: 1-shot type: bleu value: 28.30
- name: 3-shot type: bleu value: 29.26
- name: 5-shot type: bleu value: 29.27
- task:
type: text-generation
dataset:
name: WMT_RO-EN
type: WMT_RO-EN
metrics:
- name: 0-shot type: bleu value: 3.69
- name: 1-shot type: bleu value: 5.45
- name: 3-shot type: bleu value: 19.92
- name: 5-shot type: bleu value: 23.80
- task:
type: text-generation
dataset:
name: XQuAD_EM
type: XQuAD_EM
metrics:
- name: 0-shot type: exact_match value: 23.36
- name: 1-shot type: exact_match value: 47.98
- name: 3-shot type: exact_match value: 51.85
- name: 5-shot type: exact_match value: 51.43
- task:
type: text-generation
dataset:
name: XQuAD_F1
type: XQuAD_F1
metrics:
- name: 0-shot type: f1 value: 46.29
- name: 1-shot type: f1 value: 67.40
- name: 3-shot type: f1 value: 70.58
- name: 5-shot type: f1 value: 70.53
- task:
type: text-generation
dataset:
name: STS_Spearman
type: STS_Spearman
metrics:
- name: 1-shot type: spearman value: 77.91
- name: 3-shot type: spearman value: 77.73
- name: 5-shot type: spearman value: 76.65
- task:
type: text-generation
dataset:
name: STS_Pearson
type: STS_Pearson
metrics:
- name: 1-shot type: pearson value: 78.03
- name: 3-shot type: pearson value: 78.74
- name: 5-shot type: pearson value: 78.53
- task:
type: text-generation
dataset:
name: RoMT-Bench
type: RoMT-Bench
metrics:
- name: OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17
results:
Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->RoMistral is a family of pretrained and fine-tuned generative text models for Romanian. This is the repository for the instruct 7B model. Links to other models can be found at the bottom of this page.
Model Details
Model Description
<!-- Provide a longer summary of what this model is. -->OpenLLM-Ro represents the first open-source effort to build a LLM specialized for Romanian. OpenLLM-Ro developed and publicly releases a collection of Romanian LLMs, both in the form of foundational model and instruct and chat variants.
- Developed by: OpenLLM-Ro
- Language(s): Romanian
- License: cc-by-nc-4.0
- Finetuned from model: Mistral-7B-v0.1
- Trained using: RoAlpaca, RoAlpacaGPT4, RoDolly, RoSelfInstruct, RoNoRobots, RoOrca, RoCamel
Model Sources
<!-- Provide the basic links for the model. -->- Repository: https://github.com/OpenLLM-Ro/LLaMA-Factory
- Paper: https://arxiv.org/abs/2406.18266
Intended Use
Intended Use Cases
RoMistral is intented for research use in Romanian. Base models can be adapted for a variety of natural language tasks while instruction and chat tuned models are intended for assistant-like chat.
Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->Use in any manner that violates the license, any applicable laws or regluations, use in languages other than Romanian.
How to Get Started with the Model
Use the code below to get started with the model.
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("OpenLLM-Ro/RoMistral-7b-Instruct-05-17")
model = AutoModelForCausalLM.from_pretrained("OpenLLM-Ro/RoMistral-7b-Instruct-05-17")
instruction = "Ce jocuri de societate pot juca cu prietenii mei?"
chat = [
{"role": "user", "content": instruction},
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, system_message="")
inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
outputs = model.generate(input_ids=inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0]))
Academic Benchmarks
<table> <tbody> <tr> <td><strong>Model</strong></td> <td><strong><center>Average</center></strong></td> <td><strong><center>ARC</center></strong></td> <td><strong><center>MMLU</center></strong></td> <td><strong><center>Winogrande</center></strong></td> <td><strong><center>Hellaswag</center></strong></td> <td><strong><center>GSM8k</center></strong></td> <td><strong><center>TruthfulQA</center></strong></td> </tr> <tr> <td>Mistral-7B-Instruct-v0.2</td><td><center>47.40</center></td><td><center>46.29</center></td><td><center>47.00</center></td><td><center>58.78</center></td><td><center>54.27</center></td><td><center>13.47</center></td><td><center><strong>64.59</strong></center></td> </tr> <tr> <td><em>RoMistral-7b-Instruct-2024-05-17</em></td><td><center><em>52.54</em></center></td><td><center><em>50.41</em></center></td><td><center><em><strong>51.61</strong></em></center></td><td><center><em>66.48</em></center></td><td><center><em>60.27</em></center></td><td><center><em><strong>34.19</strong></em></center></td><td><center><em>52.30</em></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-2024-10-09</td><td><center><strong>52.91</strong></center></td><td><center><strong>52.27</strong></center></td><td><center>49.33</center></td><td><center><strong>70.03</strong></center></td><td><center><strong>62.88</strong></center></td><td><center>32.42</center></td><td><center>50.51</center></td> </tr> <tr> <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center>51.95</center></td><td><center>50.73</center></td><td><center>47.88</center></td><td><center>68.41</center></td><td><center>62.27</center></td><td><center>32.27</center></td><td><center>50.12</center></td> </tr> </tbody> </table>Downstream tasks
<table> <tbody> <tr> <td></td> <td colspan="4"><center><strong>LaRoSeDa</strong></center></td> <td colspan="4"><center><strong>WMT</strong></center></td> </tr> <tr> <td></td> <td colspan="2"><center><strong>Few-shot</strong></center></td> <td colspan="2"><center><strong>Finetuned</strong></center></td> <td colspan="2"><center><strong>Few-shot</strong></center></td> <td colspan="2"><center><strong>Finetuned</strong></center></td> </tr> <tr> <td><strong>Model</strong></td> <td><center><strong>Binary<br>(Macro F1)</strong></center></td> <td><center><strong>Multiclass<br>(Macro F1)</strong></center></td> <td><center><strong>Binary<br>(Macro F1)</strong></center></td> <td><center><strong>Multiclass<br>(Macro F1)</strong></center></td> <td><center><strong>EN-RO<br>(Bleu)</strong></center></td> <td><center><strong>RO-EN<br>(Bleu)</strong></center></td> <td><center><strong>EN-RO<br>(Bleu)</strong></center></td> <td><center><strong>RO-EN<br>(Bleu)</strong></center> </tr> <tr> <td>Mistral-7B-Instruct-v0.2</td><td><center>96.97</center></td><td><center>56.66</center></td><td><center>98.83</center></td><td><center>87.32</center></td><td><center>18.60</center></td><td><center><strong>33.99</strong></center></td><td><center>26.19</center></td><td><center>39.88</center></td> </tr> <tr> <td><em>RoMistral-7b-Instruct-2024-05-17</em></td><td><center><em><strong>97.36</strong></em></center></td><td><center><em>67.55</em></center></td><td><center><em>98.80</em></center></td><td><center><em><strong>88.28</strong></em></center></td><td><center><em>27.93</em></center></td><td><center><em>13.21</em></center></td><td><center><em><strong>28.72</strong></em></center></td><td><center><em><strong>40.86</strong></em></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-2024-10-09</td><td><center>95.56</center></td><td><center><strong>67.83</strong></center></td><td><center><strong>99.00</strong></center></td><td><center>87.57</center></td><td><center><strong>28.28</strong></center></td><td><center>6.10</center></td><td><center>27.70</center></td><td><center>40.36</center></td> </tr> <tr> <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center>82.13</center></td><td><center>65.24</center></td><td><center>-</center></td><td><center>-</center></td><td><center>26.25</center></td><td><center>6.09</center></td><td><center>-</center></td><td><center>-</center></td> </tr> </tbody> </table> <table> <tbody> <tr> <td></td> <td colspan="4"><center><strong>XQuAD</strong></center></td> <td colspan="4"><center><strong>STS</strong></center></td> </tr> <tr> <td></td> <td colspan="2"><center><strong>Few-shot</strong></center></td> <td colspan="2"><center><strong>Finetuned</strong></center></td> <td colspan="2"><center><strong>Few-shot</strong></center></td> <td colspan="2"><center><strong>Finetuned</strong></center></td> </tr> <tr> <td><strong>Model</strong></td> <td><center><strong>(EM)</strong></center></td> <td><center><strong>(F1)</strong></center></td> <td><center><strong>(EM)</strong></center></td> <td><center><strong>(F1)</strong></center></td> <td><center><strong>(Spearman)</strong></center></td> <td><center><strong>(Pearson)</strong></center></td> <td><center><strong>(Spearman)</strong></center></td> <td><center><strong>(Pearson)</strong></center></td> </tr> <tr> <td>Mistral-7B-Instruct-v0.2</td><td><center>27.92</center></td><td><center>50.71</center></td><td><center><strong>65.46</strong></center></td><td><center><strong>79.73</strong></center></td><td><center>62.62</center></td><td><center>60.86</center></td><td><center>84.92</center></td><td><center>85.44</center></td> </tr> <tr> <td><em>RoMistral-7b-Instruct-2024-05-17</em></td><td><center><em><strong>43.66</strong></em></center></td><td><center><em><strong>63.70</strong></em></center></td><td><center><em>55.04</em></center></td><td><center><em>72.31</em></center></td><td><center><em>77.43</em></center></td><td><center><em><strong>78.43</strong></em></center></td><td><center><em>87.25</em></center></td><td><center><em>87.79</em></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-2024-10-09</td><td><center>41.09</center></td><td><center>63.21</center></td><td><center>47.56</center></td><td><center>62.69</center></td><td><center><strong>78.47</strong></center></td><td><center>77.24</center></td><td><center><strong>87.28</strong></center></td><td><center><strong>87.88</strong></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center>23.40</center></td><td><center>45.80</center></td><td><center>-</center></td><td><center>-</center></td><td><center>77.33</center></td><td><center>76.60</center></td><td><center>-</center></td><td><center>-</center></td> </tr> </tbody> </table>MT-Bench
<table> <tbody> <tr> <td><strong>Model</strong></td> <td><strong><center>Average</center></strong></td> <td><strong><center>1st turn</center></strong></td> <td><strong><center>2nd turn</center></strong></td> <td><strong><center>Answers in Ro</center></strong></td> </tr> <tr> <td>Mistral-7B-Instruct-v0.2</td><td><center>5.03</center></td><td><center>5.05</center></td><td><center>5.00</center></td><td><center>154/160</center></td> </tr> <tr> <td><em>RoMistral-7b-Instruct-2024-05-17</em></td><td><center><em>4.99</em></center></td><td><center><em>5.46</em></center></td><td><center><em>4.53</em></center></td><td><center><em><strong>160/160</strong></em></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-2024-10-09</td><td><center>5.29</center></td><td><center>5.86</center></td><td><center>4.72</center></td><td><center><strong>160/160</strong></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center><strong>5.88</strong></center></td><td><center><strong>6.44</strong></center></td><td><center><strong>5.33</strong></center></td><td><center><strong>160/160</strong></center></td> </tr> </tbody> </table>RoCulturaBench
<table> <tbody> <tr> <td><strong>Model</strong></td> <td><strong><center>Average</center></strong></td> <td><strong><center>Answers in Ro</center></strong></td> </tr> <tr> <td>Mistral-7B-Instruct-v0.2</td><td><center>3.68</center></td><td><center>97/100</center></td> </tr> <tr> <td><em>RoMistral-7b-Instruct-2024-05-17</em></td><td><center><em>3.38</em></center></td><td><center><em><strong>100/100</strong></em></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-2024-10-09</td><td><center>3.99</center></td><td><center><strong>100/100</strong></center></td> </tr> <tr> <td>RoMistral-7b-Instruct-DPO-2024-10-09</td><td><center><strong>4.72</strong></center></td><td><center><strong>100/100</strong></center></td> </tr> </tbody> </table>RoMistral Model Family
| Model | Link |
|---|---|
| RoMistral-7b-Instruct-2024-05-17 | link |
| RoMistral-7b-Instruct-2024-10-09 | link |
| RoMistral-7b-Instruct-DPO-2024-10-09 | link |
Citation
@misc{masala2024vorbecstiromanecsterecipetrain,
title={"Vorbe\c{s}ti Rom\^ane\c{s}te?" A Recipe to Train Powerful Romanian LLMs with English Instructions},
author={Mihai Masala and Denis C. Ilie-Ablachim and Alexandru Dima and Dragos Corlatescu and Miruna Zavelca and Ovio Olaru and Simina Terian-Dan and Andrei Terian-Dan and Marius Leordeanu and Horia Velicu and Marius Popescu and Mihai Dascalu and Traian Rebedea},
year={2024},
eprint={2406.18266},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2406.18266},
}
- Created
- Oct 11, 2024
- Updated
- Oct 11, 2024