gl_Llama-3.1-8B

HiTZ/gl_Llama-3.1-8B

This is a Galician (gl) language-specific base language model trained by the HiTZ Research Center, starting from Llama 3.1 and further pretrained on curated Galician data.

This model is released as a base model, intended for further fine-tuning or adaptation (e.g., instruction tuning, domain adaptation).

---

Training Data

To train language-specific base LLMs, we followed the methodology proposed by Etxaniz et al. (2024), originally developed for Basque, and extended it to other low-resource languages. To enable fair comparisons across languages, we limited the corpus size for each language to roughly the same number of tokens. We also included a small English subset to mitigate catastrophic forgetting.

Corpus composition

Token counts vary slightly depending on the tokenizer, but remain comparable in overall size.

Data sources

Galician data was obtained from the CorpusNÓS corpus, which comprises large-scale web crawls and texts from public administrations, among other sources.
The English subset was sampled from the FineWeb corpus.

---

Model Training

• Sequence length: 8,196 tokens
• Effective batch size: 256 sequences
• Tokens per optimization step: ~2M
• Learning rate schedule: cosine decay with 10% warm-up
• Peak learning rate: 1e-5

Training was conducted on the CINECA Leonardo high-performance computing cluster using Fully Sharded Data Parallel (FSDP) across 32 nodes, each equipped with 4 NVIDIA A100 GPUs (64 GB).

---

Getting Started

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "HiTZ/gl_Llama-3.1-8B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tokenizer("Ola!", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Acknowledgements

This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and of Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project).