Downloads · 30 days
0
nyayalm-tokenizer-test
chhatramani/nyayalm-tokenizer-test
nyayalm-tokenizer-test is a machine learning model from chhatramani. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
SentencePiece-BPE tokenizer (64,000 vocab) for NyayaLM-200M, a bilingual (Nepali/English) legal-domain model with a Qwen3-style architecture and ChatML-style special tokens.
Access
Public
Updated Jul 4, 2026
Repo size
1.8 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json8.7 MB · 73%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Jul 4, 2026
- Updated
- Jul 4, 2026
- SHA
- ba11d882
Model card
From the Hugging Face model README
NyayaLM Tokenizer
SentencePiece-BPE tokenizer (64,000 vocab) for NyayaLM-200M, a bilingual (Nepali/English) legal-domain model with a Qwen3-style architecture and ChatML-style special tokens.
- Model type: BPE, byte-fallback enabled (never emits
<unk>) - Character coverage: 0.9995
- Curated special tokens: 17
- Reserved tokens for future use: 32
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("REPO_ID")
tok.apply_chat_template(
[{'role': "user", "content": "नमस्ते"}],
tokenize=False, add_generation_prompt=True,
)
- License
- apache-2.0
- Languages
- ne, en
- Created
- Jul 4, 2026
- Updated
- Jul 4, 2026