hindi-modernbert

hindi-modernBERT

TL;DR: This is a pretrained base MLM checkpoint: a Hindi extension of the ModernBERT architecture, trained from scratch on Hindi text. The base model is competitive with other models across tasks, and it outperforms them on retrieval after DPR fine-tuning. Checkpoint ba1157 · 8192 context · Hindi BPE vocab 50,368.

This release uses the ModernBERT architecture and training recipe, adapted for Hindi with a new tokenizer and ~28B tokens of Hindi pretraining.

Model summary

Eval summary

hindi-modernBERT is competitive on supervised Hindi understanding tasks and is the strongest model in this comparison on retrieval after DPR fine-tuning.

Checkpoint folders

The repository root contains the main ba1157 release. Earlier checkpoints are also available in subfolders:

Usage

Main release:

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "kkkamur07/hindi-modernbert"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

text = "भारत [MASK] विशाल देश है।"
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    logits = model(**inputs).logits

mask_idx = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
print(tokenizer.decode([logits[0, mask_idx].argmax(dim=-1).item()]))

Checkpoint subfolder:

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "kkkamur07/hindi-modernbert"
subfolder = "checkpoints/phase1"

tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder)
model = AutoModelForMaskedLM.from_pretrained(repo_id, subfolder=subfolder)

Apply Devanagari script normalisation + NFKC before tokenization for best results. See the training repo for preprocess_for_tokenizer().

For dense retrieval, fine-tune this base checkpoint with the DPR recipe on Hindi mMARCO triples. The training entrypoint is scripts/run_retrieval_finetune.py:

make retrieval-finetune ARGS="retrieval_ft.backbone=kkkamur07/hindi-modernbert retrieval_ft.max_seq_length=8192"

Evaluation

Benchmark numbers below fine-tune this base checkpoint on Naamapadam Hindi NER and MASSIVE Hindi intent. Retrieval numbers use a DPR fine-tuned checkpoint trained on Hindi mMARCO triples and evaluated on mmarco_hindi + MLDR hi.

Phase 1 → phase 2

Baseline comparison

Retrieval

Retrieval uses a DPR fine-tuned checkpoint built from this base checkpoint.

After DPR fine-tuning on 1.25M mMARCO Hindi triplets, hindi-modernBERT outperforms the other Hindi baselines on both full mMARCO Hindi and long-document MLDR hi.

The 8192-token backbone is what separates hindi-modernBERT on long-document MLDR: IndicBERTv2 is close on short mMARCO but falls behind on MLDR hi because it cannot encode full long documents. These are DPR numbers, one vector per document.

Full retrieval benchmarks (DPR, 8192 context):

Metric glossary: nDCG@10 measures ranking quality in the top 10, Recall@10 measures whether relevant passages appear in the top 10, and MRR@10 measures how high the first relevant result appears.

Full report: eval_summary_report.md.

Data

Pretraining

BPE tokenizer trained on Sangraha Hindi. MLM eval holdout: one withheld Sangraha shard (174k rows).

Downstream evaluation (Hindi fine-tune)

Transfer evaluation (source train → Hindi eval)

Retrieval fine-tuning + evaluation

Limitations

• Hindi-only pretraining: other Indic languages are not in scope for this release.
• Tokenizer preprocessing: Devanagari script normalization is required for best results and is not applied automatically by AutoTokenizer.
• Not a retriever by itself: dense retrieval requires DPR fine-tuning on top of this base checkpoint.
• Single-seed evals: downstream and retrieval benchmarks use one fine-tuning seed; multi-seed averaging may shift scores slightly.
• Phase-1 holdout: MLM eval holdout was Sangraha-only; future runs may mix holdout sources for a more representative training signal.

Acknowledgments

• AI4Bharat: Sangraha, IndicCorp V2, Naamapadam, IndicSentiment, IndicQA, IndicCOPA
• Amazon Science: MASSIVE Hindi intent
• Transfer eval sources: amazon_reviews_multi, XTREME SQuAD, Social IQa
• Unicamp DL: mMARCO Hindi DPR training triples
• AIhnIndicRag: mmarco_hindi retrieval benchmark
• BGE / MLDR: MLDR long-document retrieval benchmark
• Answer.AI / ModernBERT team: architecture (arxiv:2412.13663)

Citation

@misc{hindi-modernbert2026,
  title  = {hindi-modernBERT: A Hindi ModernBERT Encoder with 8192 Context},
  author = {Krrish Agarwalla},
  year   = {2026},
  note   = {Checkpoint ba1157. Base MLM; trained from scratch on Hindi.}
}

@article{modernbert2024,
  title  = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author = {Warner, Benjamin and Chizhov, Anton and Ermolaev, Alexander and others},
  journal = {arXiv preprint arXiv:2412.13663},
  year   = {2024}
}

License

Apache 2.0.