Downloads · 30 days
0
kimi-k2.6-nepali-tokenizer
sidskarki/kimi-k2.6-nepali-tokenizer
kimi-k2.6-nepali-tokenizer is a machine learning model from sidskarki. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
Extended tokenizer for Kimi K2.6 with ~15K added high-value Nepali/Devanagari tokens.
Access
Public
Updated May 12, 2026
Repo size
2.8 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json3.4 MB · 55%
At a glance
- License
- other
- Access
- Public
- Created
- May 9, 2026
- Updated
- May 12, 2026
- SHA
- 8df8eb88
Model card
From the Hugging Face model README
Kimi K2.6 — Nepali Extended Tokenizer
Extended tokenizer for Kimi K2.6 with ~15K added high-value Nepali/Devanagari tokens.
Token Efficiency
| Nepali tok/word | |
|---|---|
| Original Kimi K2.6 | 3.89 |
| Extended (this) | 2.51 |
| Reduction | 35.5% |
How It Was Built
- Trained a 32K SentencePiece BPE tokenizer on a 7.49GB cleaned Nepali corpus
- Selected tokens that the Kimi K2.6 base tokenizer splits into 3+ subtokens (delta vocabulary approach)
- Added ~15K high-value Nepali tokens to the base tokenizer
The extended tokenizer is a drop-in replacement for the original. To use the new tokens effectively, the model needs continued pretraining on Nepali text (see the Qwen3-4B Nepali model for a full CPT+SFT example).
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("sidskarki/kimi-k2.6-nepali-tokenizer")
tokens = tokenizer.tokenize("नेपालको राजधानी काठमाडौं हो")
print(tokens, len(tokens))
Context
Part of a 17-model Nepali tokenizer benchmark measuring the Nepali token tax across modern LLM tokenizers.
Links
- License
- other
- Languages
- ne, en
- Created
- May 9, 2026
- Updated
- May 12, 2026