Aria_7b_v2

ARIA 7B V2 is a model created by Faraday 🇫🇷 🇧🇪 The growing need of artificial intelligence tools around the world has created a run for GPU power. We decided to create an affordable model with better skills in French which can run on single GPU and reduce data bias observed in models trained mostly on english only datasets..

ARIA 7B has been trained on over 20.000 tokens of a high quality french dataset. ARIA 7B is one of the best open source models in the world avaible for this size of parameters.

Training procedure : NVIDIA A100. Thanks to NVIDIA GPU and Inception program,we have been able to train our model within less than 24 hours.

Base model : LLAMA_2-7B-CHAT-HF

We strongly believe that training models in more languages datasets can not only increase their knowledge base but also give more open analysis perspectives ,less focused visions and opinions from only one part of the world.

Contact

[email protected]

Number of Epoch : 2

Timing : Less than 24 hours

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• load_in_8bit: True
• load_in_4bit: False
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: fp4
• bnb_4bit_use_double_quant: False
• bnb_4bit_compute_dtype: float32

Framework versions

• PEFT 0.6.0.dev0