Aitana-2B-S

Aitana-2B-S

Table of Contents

<details> <summary>Click to expand</summary>

• Model description
• Intended uses and limitations
• How to use
• Training
• Evaluation
• Additional information

</details>

Model description

Aitana-2B-S is a generative language model with a decoder-only architecture. This model has been trained based on Salamandra-2B, using data in Valencian to achieve greater representation of this minority language, which is very similar to Catalan. This model has been continuously pre-trained for two epochs, processing 2.12 billion tokens throughout the training process. Due to the data sources used, the political and administrative domains are highly present in the model's register. The data has also been anonymised during pre-processing to avoid training with data that could violate people's privacy.

This model is based on Salamandra-2B as the basis for training and uses the same tokenizer.

Intended uses and limitations

Aitana-2B-S is a base model that can be used for causal language modeling, it can be used as is for text generation, although fine/instruction-tuning on specific tasks is recommended for its final use.

This language model has been trained with data in a formal register, namely related to the administrative and political domain, so it is expected that using it in text-generation tasks will produce text in this same format.

How to use

import torch
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
input_text = "Les corts valencianes han pres la decisió de"
model_id  = "gplsi/Aitana-2B-S"
tokenizer = AutoTokenizer.from_pretrained(model_id)
generator = pipeline(
    "text-generation",
    model=model_id,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
generation = generator(
    input_text,
    do_sample=True,
    top_k=10,
    eos_token_id=tokenizer.eos_token_id,
)
print(f"Result: {generation[0]['generated_text']}")

Training

Training data

The training corpus has been obtained using web scraping on public data from different sources such as the Official Gazette of the University of Alicante (BOUA), the Official Gazette of the Generalitat Valenciana (DOGV) and accurate data provided by the Valencian Courts (DSCV and DSCCV). Giving a total of 1.304 million tokens, according to the following table.

Several of the downloaded sources have already been used in the Salamandra-2B training, so the date of data collection for the previous model has been taken into account and those web pages have been scraped from that date.

Information on the datasets used for training is shown below:

• Official Bulletin of the University of Alicante (BOUA): These are the documents issued by the University of Alicante related to grants, regulations, and different resolutions of laws published periodically, specifically the Valencian version.

• Legacy Official Journal of the Generalitat Valenciana (DOGCV): This journal contains historical documents issued by the Valencian Community. These documents were initially recorded on paper and digitised with the standardisation of the digital format. They have the same subject matter as the DOGV documents but were generated between 1980 and 1997.

• Official Journal of the Generalitat Valenciana (DOGV): These documents contain official communications of the Valencian Community. They mainly deal with issuing laws, legal measures, and public sector communication. These journals were issued from 1998 to 2023.

• Valencian Parliament Diary Dataset (DSCCV) contains records from various committee meetings held in the parliament, with each meeting documented in a separate text file.

• Journal of the Valencian Parliament (DSCV): in this case, the transcripts of the different meetings held in the parliament's plenary sessions, with data from 1999 to 2022.

• University news (UN): We have news in a colloquial register from different universities that have Valencian as an official language, including the universities of Valencia, Alicante, Jaume I, and the Polytechnic University of Valencia.

• Valencian Journals (VJ): These include various types of Valencian journals with colloquial records to facilitate daily record-keeping alongside the legal and bureaucratic documents from previous records. These include a total of 10 different journals.

Training parameters

During the training of the model, a high context window was desired when generating text, so it was decided to use an input size of 2048 tokens and a minimum context window of 512 in case of truncating the input sequences. 80% of the data obtained was used for the training stage, while 20% was used during the evaluation stage. A summary of the parameters used during training can be seen in the following table:

Distributed Training Strategy

A distributed training strategy called Fully Sharded Data Parallel (FSDP) has been used. With this, the entire model has been loaded among the 4 A100s available for training with a mini-batch size of size 1 and a total gradient accumulation step of 64.

Languages

In addition to the data already used for the training of Salamandra-2B, data completely in Valencian from the sources mentioned in the previous section has been used.

Evaluation

In the following table, we can see the results obtained with different benchmarks in comparison with the model used for continuous pre-training. The results have been obtained from the model pre-trained; no instruction tuning or fine-tuning of any kind has been performed.

Valencian

Classification Benchmarks

Generation Benchmarks

Catalan

Classification Benchmarks

Generation Benchmarks

Spanish

Classification Benchmarks

Generation Benchmarks

English

Classification Benchmarks

Additional information

Author

The model has been developed by the Language and Information Systems Group (GPLSI) and the Centro de Inteligencia Digital (CENID), both part of the University of Alicante (UA), as part of their ongoing research in Natural Language Processing (NLP).

Contact

For further information, please send an email to GPLSI

License

Apache License 2.0

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of ILENIA-VIVES project with reference 2022/TL22/00215334.

Disclaimer

The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0.

Be aware that the model may have biases and/or any other undesirable distortions.

When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it) or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and, in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.

In no event shall the owner and creator of the model (GPLSI) be liable for any results arising from the use made by third parties.

Reference

@misc{gplsi-aitana-2B-S,
  author       = {Estevanell-Valladares, Ernesto L. and Yáñez-Romero, Fabio and Sepúlveda-Torres, Robiert and Consuegra-Ayala, Juan Pablo and Galeano, Santiago and Miró Maestre, María and Martínez-Murillo, Iván and Grande, Eduardo and Canal-Esteve, Miquel and Bonora, Mar and Gutierrez, Yoan and Abreu Salas, José Ignacio and Lloret, Elena and Montoyo, Andrés and Muñoz-Guillena and Palomar, Manuel},
  title        = {Aitana-2B-S: Continually pre-trained on Valencian},
  year         = {2025},
  institution  = {Language and Information Systems Group (GPLSI) and Centro de Inteligencia Digital (CENID), University of Alicante (UA)},
  howpublished = {\url{https://huggingface.co/gplsi/gplsi/Aitana-2B-S}},
  note         = {Accessed: 2025-12-12}
}

---

**Copyright © 2025 Language and Information Systems Group (GPLSI) and Centro de Inteligencia Digital (CENID),
University of Alicante (UA).
Distributed under the Apache License 2.0.**