maithili-clm

MaithiliCLM: First Causal Language Model for the Maithili Language

MaithiliCLM is a decoder-only causal language model (109.5M parameters) pretrained from scratch on MaithiliCorpus — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the first causal/generative language model ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.

Unlike encoder-only models (BERT, RoBERTa) that only support understanding tasks, MaithiliCLM enables text generation, making it suitable for creative writing, dialogue systems, summarization, and other generative applications in Maithili.

Model Architecture

Architecture Details

MaithiliCLM implements a modernized GPT-2 family architecture with the following enhancements:

• Rotary Position Embeddings (RoPE): Enables better length generalization than learned absolute positional embeddings
• RMSNorm: More efficient than LayerNorm, used in LLaMA and recent models
• SwiGLU: Improved activation function over GeLU, used in PaLM and LLaMA
• Causal Self-Attention: Standard autoregressive masking for language modeling

Tokenizer

The tokenizer is a byte-level BPE trained from scratch on MaithiliCorpus with a vocabulary of 32,000 tokens. This compact vocabulary reduces embedding parameters from 153.6M to 24.6M, shrinking the embedding layer from 55% to 22% of total model parameters.

Training

Data Preparation

The training corpus underwent rigorous cleaning and quality filtering:

1. Source: AiventraLab/maithili-corpus (49,629 raw documents)
2. Cleaning: HTML removal, encoding normalization, language verification
3. LID Filtering: Retained documents with ≥50% Maithili sentences (48,064 documents)
4. Train/Val Split: 47,103 training / 961 validation documents
5. Tokenization: Byte-level BPE encoding, packed into 1,024-token blocks

Training Configuration

• Duration: ~12 hours
• Tokens Seen: 983M (~10.3 epochs)
• Peak VRAM: 6.46 GB
• Throughput: ~26,500 tokens/second

Training Dynamics

Training plateaued after ~30K steps with diminishing returns (<1 ppl improvement per 5K steps).

Generation Quality

The model generates coherent Maithili text across diverse prompts. Generated continuations are syntactically correct and semantically plausible, though coherence degrades with longer sequences due to the limited training data.

Intended Use

• Text Generation: Creative writing, article completion, dialogue systems
• Fine-tuning Base: Adapt for classification, NER, QA, summarization
• Research: Foundation for Maithili NLP, low-resource language modeling
• Education: Teaching Maithili language and computational linguistics

How to Use

from transformers import AutoTokenizer
import torch

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("AiventraLab/maithili-clm")

# Load model (requires custom model class - see model.py)
# model = MaithiliGPT.from_pretrained("AiventraLab/maithili-clm")

# Generate text
prompt = "मैथिली भाषा एक"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Autoregressive generation
output = model.generate(
    input_ids,
    max_new_tokens=100,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    do_sample=True
)

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

Acknowledgments

The authors gratefully acknowledge AnK Komputing for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.

Citation

@article{maithili2026clm,
  title={MaithiliCLM: First Causal Language Model for the Maithili Language},
  author={[Authors]},
  journal={arXiv preprint},
  year={2026}
}

License

CC-BY 4.0 applies to the model weights, tokenizer, and documentation. The training data (MaithiliCorpus) retains its original license terms.

See Also

• MaithiliCorpus - Training data
• MaithiliBERT - Encoder-only counterpart