Downloads · 30 days
0
Kashmiri_Unigram_Tokenizer
Omarrran/Kashmiri_Unigram_Tokenizer
Kashmiri_Unigram_Tokenizer is a translation model from Omarrran. Use it when you need text moved from one language to another. It is set up for transformers. The card lists the license as apache-2.0.
A Unigram tokenizer for Kashmiri (ISO 639-3: kas) trained on KS-LIT-3M, a 3.1M-word literary corpus. Released as part of the KashTok study (Malik et al., 2026), the first systematic linguist-verified tokenization comp…
Access
Public
Updated May 7, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json3.1 KB · 41%
At a glance
- Task
- Translation
- Library
- transformers
- License
- apache-2.0
- Access
- Public
- Created
- Apr 30, 2026
- Updated
- May 7, 2026
- SHA
- 0aef401d
Model card
From the Hugging Face model README
Unigram Tokenizer for Kashmiri
A Unigram tokenizer for Kashmiri (ISO 639-3: kas) trained on
KS-LIT-3M, a 3.1M-word literary corpus. Released as part of the
KashTok study (Malik et al., 2026), the first systematic
linguist-verified tokenization comparison for Kashmiri.
Quick Start
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Omarrran/Kashmiri_Unigram_Tokenizer")
text = "کٲشِر زَبان چھِیہٕ خٲص زَبان"
encoding = tokenizer(text, return_tensors="pt")
print(tokenizer.tokenize(text))
print(encoding.input_ids)
Model Details
| Property | Value |
|---|---|
| Tokenizer type | Unigram |
| Vocabulary size | 4 |
| Training corpus | KS-LIT-3M (2.47M words, 129,672 train segments) |
| Special tokens | [PAD] [UNK] [CLS] [SEP] [MASK] |
| Max sequence length | 512 |
| Pre-tokenization | NFC + KS_CHAR_MAP normalization |
Evaluation Metrics
Computed on 16,209 truly-unseen held-out test segments from KS-LIT-3M:
| Metric | Value |
|---|---|
| Composite Quality Score (CQS) | 0.8848 |
| Fertility (↓ better) | 1.2000 |
| Diacritic Preservation Score (↑) | 0.9859 |
| Morphological Boundary Alignment (↑) | 0.3506 |
| Out-of-Vocabulary rate (↓) | 0.0000 |
| Reconstruction (char-level, ↑) | 1.0000 |
See the paper for full evaluation methodology and the linguist-verified gold morpheme reference.
Recommended Use
Primary use case: Multilingual NMT, morphology-aware tasks (highest CQS)
16K vocab — best MBA (0.39) + CQS (0.88). byte_fallback=True.
Companion Repositories
The other four KashTok tokenizers are also available for direct comparison:
- Kashmiri_Char_Tokenizer
- Kashmiri_Word_Tokenizer
- Kashmiri_WordPiece_Tokenizer
- Kashmiri_BPE_Tokenizer
- Kashmiri_Unigram_Tokenizer
Citation
@article{malik2026kashtok,
title = {KashTok: Tokenizing Kashmiri at Scale with Novel
Diacritic- and Morphology-Aware Metrics},
author = {Malik, Haq Nawaz and Nissar, Nahfid and others},
year = {2026}
}
Linguistic Verification
Every Kashmiri character, diacritic, and morpheme split used in the evaluation of this tokenizer was confirmed by a native-Kashmiri-speaker linguistic review (40 consonants, 7 vowels, 11 diacritics, 26 gold morpheme splits).
- Task
- Translation
- Library
- transformers
- License
- apache-2.0
- Languages
- ks, kas
- Created
- Apr 30, 2026
- Updated
- May 7, 2026