gervasio-70b-portuguese-ptpt-decoder-quantized-4bit

</br> </br> <img align="left" width="40" height="40" src="https://github.githubassets.com/images/icons/emoji/unicode/1f917.png"> <p style="text-align: center;">&nbsp;&nbsp;&nbsp;&nbsp;This is the model card for Gervásio 70B PTPT Decoder quantized at 4 bits. <br>This model is integrated in the <a href="https://evaristo.ai"><b>Evaristo.ai chatbot</b></a>, where its generative capabilities can be experimented with on the fly through a GUI. You may be interested in some of the other models in the <a href="https://huggingface.co/PORTULAN">Albertina (encoders), Gervásio (decoders) and Serafim (sentence encoder) families</a>. </p> </br> </br> <img width="500" src="logo_gervasio_long_color.png">

Gervásio 70B PTPT

Gervásio PT* is a fully open decoder for the Portuguese language.

It is a decoder of the LLaMA family, based on the neural architecture Transformer and developed over the LLaMA-3.3 70B instruct model. Its further improvement through additional training was done over language resources that include new instruction data sets of Portuguese prepared for this purpose (extraGLUE-Instruct , NatInst-PTPT, MMLU-PTPT,Wiki-PTPT).

Gervásio 70B PTPT is developed by NLX-Natural Language and Speech Group, at the University of Lisbon, Faculty of Sciences, Department of Informatics, Portugal.

For the record, its full name is Llama 3 é a base para Gervásio Produzir Textos em Português, to which corresponds the natural acronym GPT PT, and which is known more shortly as Gervásio PT* or, even more briefly, just with the nickname Gervásio, among its acquaintances.

Gervásio 70B PTPT is developed by a team from the University of Lisbon, Portugal.

The model in this repository is a version of Gervásio 70B PTPT quantized at 4 bits (Q4_K_M) in GGUF format. The non-quantized version can be found here.

<br>

Model Description

This model card is for Gervásio 70B PTPT quantized at 4 bit. The model has 70 billion parameters, a hidden size of 8,192 units, an intermediate size of 28,672 units, 64 attention heads, 80 hidden layers, and a vocabulary size of 128,000 tokens.

Gervásio 70B PTPT is distributed under an MIT license.

<br>

Training Data

Gervásio 70B PTPT was trained on various datasets, either native to European Portuguese or translated into European Portuguese:

Translated datasets:

We selected only datasets where the outcome of their translation into European Portuguese could preserve, in the target language, the linguistic properties at stake.

• MMLU (multiple choice question answering)

• Subset of Natural Language Instructions (multiple choice question answering)

From GLUE, we resorted to the following four tasks:

• MRPC (paraphrase Detection).
• RTE (recognizing Textual Entailment).
• STS-B (semantic textual similarity).
• WNLI (coreference and natural language inference).

And from SuperGLUE, we included these other four tasks:

• BoolQ (yes/no question answering).
• CB (inference with 3 labels).
• COPA (reasoning)
• MultiRC (question answering).

Native dataset:

• Wikipedia, Human curated subset of the Portuguese Wikipedia pertaining Portuguese history, society, and culture.

Furthermore, instruction templates have been manually crafted for each task.

Training Details

Technical report forthcoming

<!--- We applied supervised fine-tuning with a causal language modeling training objective following a zero-out technique during the fine-tuning process. Specifically, while the entire prompt received attention during fine-tuning, only the response tokens were subjected to back-propagation. In terms of hyper-parameters, the model was trained with a learning rate of 2 * 10^-5, a weight decay of 0.1, a two-epoch training regime without warm-up, and to ensure the same number of tokens back-propagated per step, we employed an input sequence of 512 tokens with a batch size of 16 and 16 accumulation steps. Due to hardware limitations that imposed a shorter sequence length (512) compared to the base model (4096), instead of the typical practice of concatenating all training examples and then dividing them into batches with the same input sequence length, we separated each example individually. In other words, each example occupies the full input sequence length. -->

Performance

For testing, we evaluate on the translated datasets MRPC (similarity) and RTE (inference), COPA (reasoning/qa), MMLU (question answering), MMLU-Pro (question answering), GPQA-diamond (question answering). The respective scores in the table below were obtained with the 16-bit version.

We also evaluate on the translated DoNotAnswer-PT (answer refusal) and on Tuguesice-PT, specifically created to assess question answering on Portuguese culture. The respective scores in the table below were obtained with the 4-bit version.

<br>

How to use

You can use this model directly with a pipeline for causal language modeling:

>>> from transformers import pipeline
>>> generator = pipeline(model='PORTULAN/gervasio-70b-portuguese-ptpt-decoder-quantized-4bit')
>>> generator("A comida portuguesa é", max_new_tokens=10)

<br>

Please cite

@misc{gervasio,
      title={Advancing Generative AI for Portuguese with
             Open Decoder Gervásio PT-*}, 
      author={Rodrigo Santos, João Silva, Luís Gomes,
              João Rodrigues, António Branco},
      year={2024},
      eprint={2402.18766},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Please use the above canonical reference when using or citing this model.

Acknowledgments

The research reported here was partially supported by: PORTULAN CLARIN—Research Infrastructure for the Science and Technology of Language, funded by Lisboa 2020, Alentejo 2020 and FCT—Fundação para a Ciência e Tecnologia under the grant PINFRA/22117/2016; research project GPT-PT - Transformer-based Decoder for the Portuguese Language, funded by FCT—Fundação para a Ciência e Tecnologia under the grant CPCA-IAC/AV/478395/2022; innovation project ACCELERAT.AI - Multilingual Intelligent Contact Centers, funded by IAPMEI, I.P. - Agência para a Competitividade e Inovação under the grant C625734525-00462629, of Plano de Recuperação e Resiliência, call RE-C05-i01.01 – Agendas/Alianças Mobilizadoras para a Reindustrialização.