Downloads · 30 days
0
nyayalm-tokenizer-64k_v3
chhatramani/nyayalm-tokenizer-64k_v3
nyayalm-tokenizer-64k_v3 is a machine learning model from chhatramani. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Domain-specific bilingual (Nepali + English) SentencePiece-BPE tokenizer for the NyayaLM-200M legal-domain language model.
Access
Public
Updated Aug 5, 2026
Repo size
1.3 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json8.3 MB · 77%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 5, 2026
- Updated
- Aug 5, 2026
- SHA
- 321b5f79
Model card
From the Hugging Face model README
NyayaLM 64K Tokenizer
Domain-specific bilingual (Nepali + English) SentencePiece-BPE tokenizer for the NyayaLM-200M legal-domain language model.
- Vocab size: 64 000
- Algorithm: SentencePiece BPE + byte_fallback
- Special tokens: Qwen3 / ChatML style (high IDs)
- EOS:
<|im_end|> - PAD:
<|endoftext|> - Chat template: Qwen3-compatible
Quick use
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("YOUR_HF_REPO")
print(tok.apply_chat_template(
[{'role': "user", "content": "नेपालको संविधान के हो?"}],
tokenize=False,
add_generation_prompt=True,
))
- License
- apache-2.0
- Languages
- ne, en
- Created
- Aug 5, 2026
- Updated
- Aug 5, 2026