Aitana-6.3B

AITANA-6.3B

<img src="https://cdn-uploads.huggingface.co/production/uploads/639873bb315923c0d5b4c883/6EPbzDJbYtyX_oS15K6jF.png" width="50%" height="50%"/>

Table of Contents

<details> <summary>Click to expand</summary>

• Model description
• Intended uses and limitations
• Demo
• How to use
• Limitations and bias
• Training
• Evaluation
• Additional information

</details>

Model description

AITANA-6.3B is a text generation model for causal language modeling with a decoder-only architecture. It has been trained from continuous pre-training based on FLOR-6.3B, with emphasis on data (listed below) in Valencian (similar to Catalan) language. Concretely, a total of 1.304 million tokens per epoch in this first version of the model and two epochs over the data. The Political and Administrative domains are highly represented in this model's version.

This model is based on FLOR-6.3B as the basis for training and uses the same tokenizer.

Intended uses and limitations

As FLOR-6.3B, AITANA-6.3B is a base model that can be used for causal language modeling, it can be used as is for text generation, although fine/instruction-tuning on specific tasks is recommended for its final use.

This language model has been trained with data in a formal register, namely related to the administrative and political domain, so it is expected that using it in text-generation tasks will produce text in this same format.

Demo

In the following link, you can access an interactive demo to test the text generation in the language model:

Demo link(https://llm-aitana.gplsi.es/)

In the demo, you can adjust the number of words generated as well as the decoding technique to be used by the model (top p, top k) and other parameters such as temperature.

How to use

import torch
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

input_text = "Les corts valencianes han pres la decisió de"

model_id  = "gplsi/Aitana-6.3B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
generator = pipeline(
    "text-generation",
    model=model_id,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
generation = generator(
    input_text,
    do_sample=True,
    top_k=10,
    eos_token_id=tokenizer.eos_token_id,
)

print(f"Result: {generation[0]['generated_text']}")

Training

Training data

The training corpus has been obtained using web scraping on public data from different sources such as the Official Gazette of the University of Alicante (BOUA), the Official Gazette of the Generalitat Valenciana (DOGV) and accurate data provided by the Valencian Courts (DSCV and DSCCV). Giving a total of 1.304 million tokens, according to the following table.

Several of the downloaded sources have already been used in the FLOR-6.3B training, so the date of data collection for the previous model has been taken into account and those web pages have been scraped from that date.

Information on the datasets used for training is shown below:

• BOUA: Official Bulletin of the University of Alicante. In this case, we are dealing with documents issued by the University of Alicante in Valencian about grants, calls issued by the university, regulations, resolutions of laws that affect the university environment, and corrections of errors of these same documents issued previously.

• DOGV: Official Journal of the Generalitat Valenciana. This dataset contains official communiqués of different kinds issued by the Generalitat Valenciana, with data entirely in Valencian. It mainly talks about measures taken in the legal field, approval of laws, and public sector communiqués. In this case, we have 18 different documents covering communiqués from 1998 to 2018 and three more recent documents with data from 2019 to 2023.

• DOGCV: in this case, it is the Official Journal of the Generalitat Valenciana, but only the historical documents from 1980 to 1997.

• DSCV: Journal of the Valencian Parliament. This dataset contains transcriptions of the different interventions made during the plenary sessions in the Valencian Parliament by the different participants. It covers data from 2001 to 1999 up to 2022, each transcript comprises a .html file.

• DSCCV: this is a dataset of the Valencian Parliament diary, centered on transcriptions of the different commissions held. As in the previous case, it is separated into one file for each transcription.

Training parameters

During the training of the model, a high context window was desired when generating text, so it was decided to use an input size of 2048 tokens and a minimum context window of 512 in case of truncating the input sequences. 80% of the data obtained was used for the training stage, while 20% was used during the evaluation stage. A summary of the parameters used during training can be seen in the following table:

Devices

A total of 4 A100 graphics cards with a maximum capacity of 40 GB each were used to train the model. This meant a training time of approximately 40 hours per epoch. Using a mini-batch size of size 2 and a batch size of size 32 to calculate backpropagation.

Distributed Training Strategy

A distributed training strategy called Fully Sharded Data Parallel (FSDP) has been used. With this, the entire model has been loaded among the 4 A100s available for training with a mini-batch size of size 2 as previously discussed.

Languages

In addition to the data already used for the training of FLOR-6.3B, data completely in Valencian from the sources mentioned in the previous section has been used.

Evaluation

The model has been evaluated using the loss function and perplexity during the training stage and these metrics have also been obtained during the evaluation stage. Due to the low amount of data, it was decided to evaluate at the end of each epoch.

Results

In the following table, we can see the results obtained with different benchmarks in comparison with the model used for continuous pre-training. The results have been obtained from the model pre-trained; no instruction tuning or fine-tuning of any kind has been performed.

Additional information

Author

Language and Information System Group GPLSI

Contact

For further information, please send an email to GPLSI

Copyright

Copyright(c) 2024 by GPLSI(https://gplsi.dlsi.ua.es/).

License

Apache License 2.0

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of ILENIA-VIVES project with reference 2022/TL22/00215334.

Disclaimer

The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0.

Be aware that the model may have biases and/or any other undesirable distortions.

When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it) or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and, in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.

In no event shall the owner and creator of the model (GPLSI) be liable for any results arising from the use made by third parties.

Reference

@misc{gplsi-Aitana-6.3B,
  author       = {Sepúlveda-Torres, Robiert and Estevanell-Valladares, Ernesto L. and Yáñez-Romero, Fabio and Martínez-Murillo, Iván and Grande, Eduardo and Gutierrez, Yoan and Abreu Salas, José Ignacio and Lloret, Elena and Montoyo, Andrés and Muñoz-Guillena, Rafael and Palomar, Manuel},
  title        = {Aitana-6.3B: Base model for Valencian, Spanish and English},
  year         = {2024},
  institution  = {Language and Information Systems Group (GPLSI) and Centro de Inteligencia Digital (CENID), University of Alicante (UA)},
  howpublished = {\url{https://huggingface.co/gplsi/Aitana-6.3B}},
  note         = {Accessed: 2024-07-11}
}