khasibert

KhasiBERT: Foundational Language Model for Khasi

KhasiBERT is a foundational language model for the Khasi language, trained on 3.6 million sentences using the RoBERTa architecture. This model serves as the foundation for all downstream Khasi NLP tasks including text classification, sentiment analysis, question answering, and language generation.

Model Overview

Model Architecture

KhasiBERT follows the RoBERTa-base architecture with the following specifications:

Training Methodology

Training Objective

KhasiBERT was trained using Masked Language Modeling (MLM), where 15% of input tokens are randomly masked and the model learns to predict these masked tokens based on bidirectional context.

Training Configuration

Tokenization

A custom Byte-Level BPE tokenizer was trained specifically on the Khasi corpus with:

• Vocabulary size: 32,000 tokens
• Special tokens: <s>, </s>, <pad>, <unk>, <mask>
• Trained on the complete Khasi dataset for optimal language coverage

Training Data

Dataset Statistics

Data Preprocessing

• Exact duplicate removal
• Near-duplicate removal (80% similarity threshold)
• Length filtering (10-500 characters)
• Text normalization and cleaning
• Quality validation

Usage

Quick Start

from transformers import RobertaForMaskedLM, RobertaTokenizerFast, pipeline

# Load model and tokenizer
model = RobertaForMaskedLM.from_pretrained('MWirelabs/khasibert')
tokenizer = RobertaTokenizerFast.from_pretrained('MWirelabs/khasibert')

# Create fill-mask pipeline
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)

# Example usage
text = 'Ka Meghalaya ka <mask> ha ka jingpyrkhat jong ki Khasi.'
results = fill_mask(text)
print(results)

Fine-tuning for Downstream Tasks

from transformers import RobertaForSequenceClassification

# For text classification
model = RobertaForSequenceClassification.from_pretrained(
    'MWirelabs/khasibert',
    num_labels=2  # Adjust for your task
)

# Fine-tune for sentiment analysis, document classification, etc.

Model Performance

Masked Language Modeling Results

KhasiBERT demonstrates strong contextual understanding in Khasi:

Key Strengths

• Authentic Khasi understanding with contextually appropriate predictions
• Pronoun recognition - correctly predicts "phi" (you) in gratitude expressions (44.7%)
• Semantic relationships - predicts "jingthied" related to "thied" (25.3%)
• Place name recognition - identifies proper nouns like "Delhi" (10.6%)
• Grammatical structure awareness across question, location, and action contexts

Applications

KhasiBERT serves as the foundation for various Khasi NLP applications:

Supported Tasks

• Text Classification: Document categorization, topic modeling
• Sentiment Analysis: Opinion mining in Khasi text
• Named Entity Recognition: Person, place, organization extraction
• Question Answering: Khasi reading comprehension systems
• Text Generation: Coherent Khasi text creation
• Language Understanding: Chatbots and virtual assistants
• Machine Translation: English-Khasi translation systems

Use Cases

• Educational technology for Khasi language learning
• Government services in Meghalaya
• Cultural preservation and digital humanities
• Social media monitoring and analysis
• Content recommendation systems

Regional Significance

For Meghalaya State

• Digital Government Services: Enables Khasi language interfaces for e-governance
• Educational Technology: Powers AI-driven Khasi language learning platforms
• Cultural Preservation: Digitally preserves and promotes Khasi linguistic heritage
• Economic Development: Creates foundation for local language tech industry

For Northeast India

• Linguistic Diversity: Supports AI development for Northeast India's rich language landscape
• Digital Inclusion: Ensures indigenous communities aren't left behind in AI revolution
• Research Hub: Positions Meghalaya as center for indigenous language AI research

Technical Requirements

Hardware Requirements

Software Requirements

• Python 3.8+
• PyTorch 1.9+
• Transformers 4.20+
• CUDA 11.0+ (for GPU)

Research Context

Significance

KhasiBERT represents a significant advancement in low-resource NLP:

• A foundational model for the Khasi language
• Enables NLP research for 1.4+ million Khasi speakers
• Preserves linguistic heritage through AI technology
• Demonstrates efficient training methodology for resource-constrained scenarios

Limitations

• Trained on 1 epoch (future versions may benefit from additional training)
• Performance may vary on highly domain-specific text
• Requires task-specific fine-tuning for optimal performance
• May not capture all dialectal variations of Khasi

Citation

If you use KhasiBERT in your research or applications, please cite:

@misc{khasibert2025,
  title        = {KhasiBERT v1.0: A Foundational Language Model for Khasi},
  author       = {MWire Labs},
  year         = {2025},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17063992},
  url          = {https://doi.org/10.5281/zenodo.17063992}
}

Contact & Support

• Organization: MWirelabs
• Model Repository: https://huggingface.co/MWirelabs/khasibert
• Issues: Please report issues through the Hugging Face model page

Acknowledgments

• Training conducted on NVIDIA RTX A6000 GPU
• Built using the Transformers library by Hugging Face
• Inspired by the success of foundational models for major languages
• Dedicated to the preservation and advancement of the Khasi language

License

This model is released under the Creative Commons BY-NC 4.0 License. You are free to:

• Use for non-commercial research and education
• Modify and distribute for non-commercial purposes
• Create derivative works for research

Commercial Use: Contact MWirelabs for commercial licensing agreements. Attribution Required: Please provide appropriate credit to MWirelabs when using this model.

---

KhasiBERT: Bridging traditional Khasi language with modern AI technology.