zefiro-7b-dpo-ITA

<img src="https://hoodie-creator.s3.eu-west-1.amazonaws.com/2f7c7e41-original.png" alt="zefiro" border="0" width="400px">

Model Card for zefiro-7b-dpo-ITA

Last Update: 20/02/2024<br>

<!-- Provide a quick summary of what the model is/does. -->

Zefiro is a DPO fine tuned model for the Italian language based on Zefiro-7b-sft-ita. The model can be used as base model for more specific conversationl tasks for Italian language

Model Details

Zefiro DPO is a porting of the Zephyr model to the italian language using the wonderful recipes from alignment-handbook . It has also taken ispiration and insights from the Llamantino model developed by Università di Bari. For the implementation we combined different approaches from the two models mentioned but also from the wondeful communtity of open source.

Model description

• Model type: A 7B parameter GPT-like model fine-tuned on a mix of publicly available, synthetic datasets.
• Language(s) (NLP): Primarily Italian
• License: Apache 2
• Finetuned from model: Zefiro-7b-sft-ita
• Developed by: giux78
• Funded by: Business Operating System

Code

Can be reproduced running this colab.

Computation

It has been trained on one A100 from seeweb.it who sponsered the training. I strongly suggest is one of the cheapest and solid GPUs provider.

Evaluations:

Intended uses & limitations

The model was initially fine-tuned on a filtered and preprocessed version of ultrafeedback-preferences-ITA.

Here's how you can run the model using Transformers from 🤗 :

# Install transformers from source - only needed for versions <= v4.34
# pip install git+https://github.com/huggingface/transformers.git
# pip install accelerate
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mii-community/zefiro-7b-dpo-ITA"
model = AutoModelForCausalLM.from_pretrained(model_id)
model.to('cuda')
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")


sys_prompt = "Sei un assistente disponibile, rispettoso e onesto. " \
         "Rispondi sempre nel modo piu' utile possibile, pur essendo sicuro. " \
         "Le risposte non devono includere contenuti dannosi, non etici, razzisti, sessisti, tossici, pericolosi o illegali. " \
         "Assicurati che le tue risposte siano socialmente imparziali e positive. " \
         "Se una domanda non ha senso o non e' coerente con i fatti, spiegane il motivo invece di rispondere in modo non corretto. " \
         "Se non conosci la risposta a una domanda, non condividere informazioni false."

messages = [{ 'content' : sys_prompt, 'role' : 'assistant'}, 
            {'content' : 'Crea una lista su cosa mangiare a pranzo ogni giorno della settimana a pranzo e cena', 'role' : 'user'}]


def generate_text(sys_prompt, user_prompt):
    messages = [{ 'content' : sys_prompt, 'role' : 'assistant'}, 
            {'content' : user_prompt, 'role' : 'user'}]
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
    generated_ids = model.generate(**model_inputs, max_new_tokens=1024)
    return tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]


generate_text(sys_prompt, 'cosa ne pensi della politica italiana?')

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

Zefiro-7b-sft-ITA has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). It is also unknown what the size and composition of the corpus was used to train the base model (mistralai/Mistral-7B-v0.1), however it is likely to have included a mix of Web data and technical sources like books and code. See the Falcon 180B model card for an example of this.

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

We used UltraChat-ITA as training data that is a filtered version of the UltraChat. For translating the dataset we combined different tools and API we are also evaluating the best approach for translating many more datasets. We have seen that the translation phase is critical and can introduce incorrect syntax and semantics.

Summary

Zefiro-7b-beta-ITA-v0.1 is finetuned version of mistral-7b using the zephyr approach for the italian language.

Citation

@misc{tunstall2023zephyr,
      title={Zephyr: Direct Distillation of LM Alignment}, 
      author={Lewis Tunstall and Edward Beeching and Nathan Lambert and Nazneen Rajani and Kashif Rasul and Younes Belkada and Shengyi Huang and Leandro von Werra and Clémentine Fourrier and Nathan Habib and Nathan Sarrazin and Omar Sanseviero and Alexander M. Rush and Thomas Wolf},
      year={2023},
      eprint={2310.16944},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

@misc{basile2023llamantino,
      title={LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language}, 
      author={Pierpaolo Basile and Elio Musacchio and Marco Polignano and Lucia Siciliani and Giuseppe Fiameni and Giovanni Semeraro},
      year={2023},
      eprint={2312.09993},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Model Card Authors

giux78

Model Card Contact

**[email protected]