maithili-roberta

MaithiliBERT: First RoBERTa Model for the Maithili Language

MaithiliBERT is a RoBERTa-base language model (124M parameters) pretrained from scratch on MaithiliCorpus — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the first RoBERTa ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.

Model Architecture

Tokenizer

The tokenizer is a byte-level BPE trained from scratch on MaithiliCorpus with a vocabulary of 50,000 tokens — 60% smaller than mBERT (119K) and 80% smaller than XLM-R/IndicBERTv2 (250K). This reduces embedding parameters from 192M to 38.4M, shrinking the embedding layer from 61% to 31% of total model parameters.

Acknowledgments

The authors gratefully acknowledge AnK Komputing for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.

Intended Use

• Masked language modeling (fill-mask)
• Fine-tuning for text classification, NER, and other NLU tasks
• Feature extraction for downstream Maithili NLP

Usage

from transformers import pipeline
fill = pipeline("fill-mask", model="AiventraLab/maithili-roberta")
results = fill("भगवान् <mask>क ध्यान")