MrBERT-nos-gl

MrBERT-nos-gl

MrBERT-nos-gl is a domain-adapted encoder model obtained by continued pre-training of BSC-LT/MrBERT on CorpusNÓS, a large-scale Galician corpus (~1.9B tokens). It inherits MrBERT's ModernBERT architecture — with efficient long-context modeling (up to 1,024 tokens) via RoPE and sliding-window attention — and extends its coverage of Galician and Portuguese, two closely related Iberian languages underrepresented in most multilingual encoders.

The model is designed as a general-purpose encoder suitable for fine-tuning on downstream tasks such as named entity recognition, part-of-speech tagging, text classification, semantic similarity, question answering, and cross-lingual retrieval. It is the foundation for the MrBERT-nos-gl model collection.

Developed as part of Proxecto Nós, an initiative to build language technology for the Galician language.

Technical Description

MrBERT-nos-gl starts from the MrBERT base checkpoint and continues pre-training with a masked language modelling objective on a combined Galician and Portuguese corpus. The architecture is identical to the base model; only the training regime and data distribution differ.

Model Architecture

Continued Pre-training Hyperparameters

Usage

This model is a base encoder intended for fine-tuning, not for direct text generation. Load it with the fill-mask pipeline for masked language modeling, or use it as a backbone for downstream task fine-tuning.

Installation

pip install transformers torch

Masked language modeling

from transformers import pipeline

mlm = pipeline("fill-mask", model="proxectonos/MrBERT-nos-gl")

results = mlm("A lingua galega é unha das linguas [MASK] de Europa.")
for r in results:
    print(f"{r['token_str']:<20} {r['score']*100:.1f}%")

Feature extraction / embeddings

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("proxectonos/MrBERT-nos-gl")
model = AutoModel.from_pretrained("proxectonos/MrBERT-nos-gl")

inputs = tokenizer("A lingua galega é unha das linguas romances de Europa.", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# Mean-pool the last hidden state for a sentence embedding
embeddings = outputs.last_hidden_state.mean(dim=1)

Acknowledgements

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA. (Esta publicación del proyecto Desarrollo de Modelos ALIA está financiada por el Ministerio para la Transformación Digital y de la Función Pública y por el Plan de Recuperación, Transformación y Resiliencia – Financiado por la Unión Europea – NextGenerationEU)