Downloads · 30 days
0
nyayalm-tokenizer-64k_v1
chhatramani/nyayalm-tokenizer-64k_v1
nyayalm-tokenizer-64k_v1 is a machine learning model from chhatramani. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
SentencePiece-BPE tokenizer (64,000 vocab) for NyayaLM-200M, bilingual (Nepali/English) legal-domain model with Qwen3-style ChatML special tokens.
Access
Public
Updated Jul 27, 2026
Repo size
2.7 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json8.3 MB · 76%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Jul 27, 2026
- Updated
- Jul 27, 2026
- SHA
- 47596d84
Model card
From the Hugging Face model README
NyayaLM Tokenizer
SentencePiece-BPE tokenizer (64,000 vocab) for NyayaLM-200M, bilingual (Nepali/English) legal-domain model with Qwen3-style ChatML special tokens.
- BPE + byte_fallback
- Character coverage: 0.9995
- Curated specials at high ids (not colliding with BPE pieces)
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("REPO_ID")
print(tok.apply_chat_template(
[{"role": "user", "content": "नमस्ते"}],
tokenize=False, add_generation_prompt=True,
))
- License
- apache-2.0
- Languages
- ne, en
- Created
- Jul 27, 2026
- Updated
- Jul 27, 2026