deberta-xsmall-gl

DeBERTa-xsmall-gl

DeBERTa-xsmall-gl is a continued pretraining checkpoint based on microsoft/deberta-v3-xsmall, adapted to Galician through large-scale masked-language modeling. It is intended as a strong general-purpose encoder for downstream NLP tasks in Galician.

Training

• Base model: microsoft/deberta-v3-xsmall
• Epochs: 3
• Learning rate: 6e-4
• MLM probability: 0.15
• Max sequence length: 512
• Total batch size: 1024
• Training examples: 6,139,791

Intended uses

• Masked language modeling (fill-mask)
• Encoder for classification, NER, QA, and general Galician NLP tasks
• Further domain adaptation via fine-tuning

How to use

from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline

model_id = "proxectonos/deberta-xsmall-gl"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)

fill_mask("O Parlamento de Galicia aprobou a <mask> hoxe.")

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA

Citation

Please reference this model as: mdeberta-gl (Proxecto Nós Team, 2025).