Vigogne_Llama-3.2-1B

Model Card: French-Continued Pretrained Language Models

Overview

Vigogne is a series of pretrained LLMs built by Zaion, leader in Conversational AI designed for CX.

This model card documents the continual pretraining of five language models on French data to enhance their proficiency in the French language:

• Llama-3.2-1B
• Llama-3.2-3B
• Llama-3.2-8B
• Qwen2.5-1.5B
• Qwen2.5-3B

Training Procedure

The training process consisted of three distinct phases:

Phase 1: Initial Pretraining

• Data Source: The French subset of the FineWeb-2 corpus.
• Learning Rate: A constant learning rate was used throughout this phase.

Phase 2: Annealing Phase

• Learning Rate Scheduler: A cosine scheduler was applied to gradually adjust the learning rate.
• Data Composition:
  • Subset of FineWeb-2: A portion of the French FineWeb-2 corpus used in Phase 1.
  • LLM-Rewritten Subset: A portion of the corpus rewritten using a Large Language Model (LLM).
  • French Magpie Dataset: A dataset specifically curated for this work, containing response components from a French magpie dataset. Available here.

Phase 3: Supervised Fine-Tuning (SFT)

In this phase, the annealed models were fine-tuned on meticulously curated in-house instruction data.

Evaluation Results

The models were evaluated on various French language tasks, with results detailed in the table below:

Reproducing Results

To replicate these results, install lm-evaluation-harness and run the following command:

lm_eval --model hf --model_args pretrained=MODEL --tasks TASK --batch_size auto

where TASK is one of the following: french_bench_arc_challenge, french_bench_grammar, french_bench_hellaswag, french_bench_boolqa, french_bench_reading_comp or french_bench_vocab.

Limitations

• The models underwent only Supervised Fine-Tuning (SFT) as post-training. Further improvements could be made using additional post-training techniques such as Direct Preference Optimization (DPO). We leave this for future work.
• The models are of limited capacity and might generate harmful or biased content, incorrect information or generally unhelpful answers.

Ethical Considerations

Users should be aware of potential biases present in the training data, which may influence model outputs. It is recommended to deploy these models responsibly, especially in sensitive applications where fairness and accuracy are crucial.

Acknowledgement

This work was granted access to the HPC resources of IDRIS under the allocation 2024-GC011015467 made by GENCI.