Downloads · 30 days
0
qwen3-nepali-tokenizer
sidskarki/qwen3-nepali-tokenizer
qwen3-nepali-tokenizer is a machine learning model from sidskarki. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
Extended tokenizer for Qwen 3 with ~15K added high-value Nepali/Devanagari tokens.
Access
Public
Updated May 12, 2026
Repo size
14.5 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json14.5 MB · 100%
At a glance
- License
- other
- Access
- Public
- Created
- May 9, 2026
- Updated
- May 12, 2026
- SHA
- d44a353d
Model card
From the Hugging Face model README
Qwen 3 — Nepali Extended Tokenizer
Extended tokenizer for Qwen 3 with ~15K added high-value Nepali/Devanagari tokens.
Token Efficiency
| Nepali tok/word | |
|---|---|
| Original Qwen 3 | 6.10 |
| Extended (this) | 3.18 |
| Reduction | 48.1% |
How It Was Built
- Trained a 32K SentencePiece BPE tokenizer on a 7.49GB cleaned Nepali corpus
- Selected tokens that the Qwen 3 base tokenizer splits into 3+ subtokens (delta vocabulary approach)
- Added ~15K high-value Nepali tokens to the base tokenizer
The extended tokenizer is a drop-in replacement for the original. To use the new tokens effectively, the model needs continued pretraining on Nepali text (see the Qwen3-4B Nepali model for a full CPT+SFT example).
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("sidskarki/qwen3-nepali-tokenizer")
tokens = tokenizer.tokenize("नेपालको राजधानी काठमाडौं हो")
print(tokens, len(tokens))
Context
Part of a 17-model Nepali tokenizer benchmark measuring the Nepali token tax across modern LLM tokenizers.
Links
- License
- other
- Languages
- ne, en
- Created
- May 9, 2026
- Updated
- May 12, 2026