Carvalho_pt-gl-1.3B

Carvalho_pt-gl-1.3B

Carvalho_pt-gl-1.3B is a 1.3B-parameter transformer-based causal language model for Galician and European Portuguese. It is the result of a continual pretraining of a Cerebras-GPT-1.3B adapted to catalan, spanish and english previously by the AINA Project.

This model is part of the Carvalho familily, a family of LLMs specialized in Portuguese and Galician which can be found here.

How to use

import torch
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

token_HF=""#Obter na páxina de HuggingFace
input_text = "Hoxe fai un bo día. O sol  "

model_id  = "Nos-PT/Carvalho_pt-gl-1.3B"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=token_HF)
model = AutoModelForCausalLM.from_pretrained(model_id, use_auth_token=token_HF)
generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
generation = generator(
    input_text,
    do_sample=True,
    top_k=10,
    eos_token_id=tokenizer.eos_token_id
)

print(f"Result: {generation[0]['generated_text']}")

Training

Tools

It was trained using HuggingFace Transformers and Pytorch, using the Causal Modeling Language script. We also use DeepSpeed to deal with the huge size of the model.

Training data

The training corpus consists of 5B tokens of texts in Galician and European Portuguese, whose main sources are CorpusNÓS for Galician and Arquivo.pt for Portuguese.

Training hyperparameters

• seed: 42
• train_batch_size: 4
• eval_batch_size: 4
• gradient_acummulation: 8
• optimizer: AdamW
• betas: (0.9,0.999)
• epsilon: 1e-08
• weight_decay_rate: 0.1
• scheduler: "Linear"
• learning_rate: 5e-05
• num_epochs: 1.0

Framework

The training was conducted on the Galician Supercomputing Center (CESGA), using various nodes with 2 GPUs NVIDIA A100 40G.

Evaluation

The evaluation was carried out using the Calame dataset, in its Portuguese and Galician versions, and in the GLUE benchmark. In the latter case, the evaluated models were fine-tuned to rotate the tasks.

Additional information

Funding

This model was development within the Nós Project, funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ILENIA with reference 2022/TL22/00215336.

Cite this model

@inproceedings{gamallo2024galician,
  title={A Galician-Portuguese Generative Model},
  author={Gamallo, Pablo and Rodr{\'\i}guez, Pablo and Santos, Daniel and Sotelo, Susana and Miquelina, Nuno and Paniagua, Silvia and Schmidt, Daniela and de-Dios-Flores, Iria and Quaresma, Paulo and Bardanca, Daniel and others},
  booktitle={EPIA Conference on Artificial Intelligence},
  pages={292--304},
  year={2024},
  organization={Springer}
}