nyayalm-SentencePiece_BPE-64k

NyayaLM 64K SentencePiece-BPE Tokenizer

A production-grade 64 000-vocab SentencePiece BPE tokenizer trained for Nepali (Devanagari), Romanized Nepali, and English, with special focus on legal and institutional text.

This is the SentencePiece_BPE tokenizer, trained on the exact same corpus and vocabulary budget for fair comparison with BPE only Tokenizer.

---

Highlights

---

Quick start

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("chhatramani/nyayalm-SentencePiece_BPE-64k")

text = "नेपालको संविधान २०७२ ले मौलिक हकको ग्यारेन्टी गर्दछ।"
ids = tok.encode(text, add_special_tokens=False)
print(len(ids), tok.decode(ids))

messages = [
    {"role": "system", "content": "You are a helpful legal assistant for Nepal."},
    {"role": "user", "content": "नेपालको संविधानमा मौलिक हक के-के छन्?"},
]
print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))

---

Citation

@misc{nyayalm_tokenizer_64k_bpe,
  title        = {NyayaLM 64K SentencePiece-BPE Tokenizer for Nepali, Romanized Nepali, and English},
  author       = {Chhatramani Yadav},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/chhatramani/nyayalm-SentencePiece_BPE-64k}},
}

---

License

Apache License 2.0.