meitei-roberta

Meitei-RoBERTa-Base (Monolingual, Bengali Script)

The Meitei-RoBERTa-Base model is a high-performance, monolingual transformer encoder pre-trained from scratch on the entire Meitei Monolingual Corpus (MWirelabs/meitei-monolingual-corpus). It is developed using the RoBERTa training methodology, establishing a foundational language representation for Meitei (Manipuri) in Bengali script.

This model serves as a robust backbone for accelerating downstream NLP tasks such as Named Entity Recognition (NER), Sentiment Analysis, and Text Classification in Meitei.

Model Architecture & Details

The architecture is based on the highly effective RoBERTa Base configuration, ensuring maximum performance while managing computational complexity.

Configuration

Tokenizer Details

Pre-training Details

The model was trained from a randomly initialized state on the full corpus using the Masked Language Modeling (MLM) objective.

Training Parameters

Training Metrics & Loss Curve

The model demonstrates strong convergence, successfully learning the grammatical and semantic structure of Meitei.

The full log history, including training loss and learning rate evolution, is available in the repository for detailed analysis:

• Log File: training_loss_history.csv

• Metric: Training Loss (loss) against Training Step (step).

---

Evaluation Metrics

Comparative Performance

The full log history, including training loss and learning rate evolution, is available in the repository for detailed analysis:

• Log File: training_loss_history.csv

---

💡 How to Use (For Inference and Fine-tuning)

This model can be loaded directly into any Hugging Face pipeline or used as the encoder in a custom model for fine-tuning.

from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline

# Using the lowercase repository ID for robust loading
REPO_ID = "MWirelabs/meitei-roberta" 

# 1. Load Model and Tokenizer
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForMaskedLM.from_pretrained(REPO_ID)

# 2. Example: Tokenize Text
# NOTE: Example text MUST be in the Bengali script, as the model was trained only on this script.
meitei_text = "আমি গতকাল স্কুল থেকে ফিরেছি। এই বইটি পড়তে ভাল লাগে।" 
inputs = tokenizer(meitei_text, return_tensors="pt")

# 3. Example: Fill-Mask Pipeline Test (Testing fluency)
unmasker = pipeline("fill-mask", model=model, tokenizer=tokenizer)
# Masking a word in Bengali script Meitei
results = unmasker("আমাদের দেশে <mask> অনেক সমস্যা আছে।") 
# ... process results

Intended Use

This model is primarily intended for research purposes and as a foundational encoder for Meitei NLP downstream tasks, including:

Fine-tuning on sequence classification, token classification (NER), and summarization.

Feature extraction to generate high-quality Meitei text embeddings. The model is not intended for deployment in applications that require safety-critical decision-making without further domain-specific fine-tuning and validation.

Limitations and Bias

Script Dependence: This model was exclusively trained on the Bengali script version of Meitei and will perform poorly on Meitei text written in the Meitei Mayek (Meetei Mayek) script.

Monolingual Focus: The model is not suitable for cross-lingual tasks without further fine-tuning.

📚 Citation

If you use this model or the Meitei corpus in your work, please cite it as:

@inproceedings{nyalang-2026-beyond,
  title     = {Beyond Multilinguality: Typological Limitations in Multilingual Models for Meitei Language},
  author    = {Nyalang, Badal},
  booktitle = {Proceedings of the 8th Workshop on Research in Computational Linguistic Typology and Multilingual NLP},
  pages     = {32--38},
  year      = {2026},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2026.sigtyp-main.5}
}

About MWire Labs

MWire Labs builds ethical, region-first AI infrastructure for Northeast India—focusing on low-resource languages and public accessibility.

Learn more at www.mwirelabs.com

---

Contributions & Feedback

We welcome feedback, contributions, and civic collaborations.
Reach out via Hugging Face.