stellialm_smallfr_qwen7b_9tplus

Stellialm small public

This model is a public version of our model.

Stellia small size model trained to perform 9 specific task in english and french with high accuracy. And an extended training in french language on instruction data. The goal is to develop an efficient model on specific task needing reasonning and very strict output format, along general instruction in french language.

The benchmark bellow is done with our evaluation pipeline.

Model Fine-tune from Qwen/Qwen2.5-7B-Instruct with a specific LoRA adapter.

The prompt of our 9 specific task might follow in a next post. The goal of our team is to specialized small LLM on our clients specific needs while keeping the highest quality on general tasks.

Our 7b models almost reach GPT-4o performances on our specific tasks according to our evaluation pipeline, an human preference evaluation is needed.

+----------------------+---------+------------+------------+
| 9tasks + general fr  | Overall | Team score | loads fail |
+----------------------+---------+------------+------------+
| answer_reformulation |   0.74  |    0.72    |    0.99    |
| query_reformulation  |   0.85  |    0.99    |     0      |
|    summarization     |   0.94  |    ---     |     1      |
|  keyword_extraction  |   0.8   |    ---     |     1      |
|  fill_in_generation  |   0.91  |    ---     |     1      |
|    keyword_update    |   0.71  |    0.92    |     5      |
|         gqa          |   0.83  |    0.65    |     2      |
|      true_false      |   0.78  |    ---     |     1      |
|         mcq          |   0.89  |    ---     |     1      |
|        Total         |   0.83  |    ---     |   12.99    |
+----------------------+---------+------------+------------+