Downloads · 30 days
0
Kashmiri_BPE_Tokenizer
Omarrran/Kashmiri_BPE_Tokenizer
Kashmiri_BPE_Tokenizer is a machine learning model from Omarrran. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
A BPE tokenizer for Kashmiri (ISO 639-3: kas) trained on KS-LIT-3M, a 3.1M-word literary corpus. Released as part of the KashTok study (Malik et al., 2026), the first systematic linguist-verified tokenization comparis…
Access
Public
Updated Apr 30, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.3 MB · 100%
At a glance
- Library
- transformers
- License
- apache-2.0
- Access
- Public
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026
- SHA
- b63957d4
Model card
From the Hugging Face model README
BPE Tokenizer for Kashmiri
A BPE tokenizer for Kashmiri (ISO 639-3: kas) trained on
KS-LIT-3M, a 3.1M-word literary corpus. Released as part of the
KashTok study (Malik et al., 2026), the first systematic
linguist-verified tokenization comparison for Kashmiri.
Quick Start
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Omarrran/Kashmiri_BPE_Tokenizer")
text = "کٲشِر زَبان چھِیہٕ خٲص زَبان"
encoding = tokenizer(text, return_tensors="pt")
print(tokenizer.tokenize(text))
print(encoding.input_ids)
Model Details
| Property | Value |
|---|---|
| Tokenizer type | BPE |
| Vocabulary size | 16,000 |
| Training corpus | KS-LIT-3M (2.47M words, 129,672 train segments) |
| Special tokens | [PAD] [UNK] [CLS] [SEP] [MASK] |
| Max sequence length | 512 |
| Pre-tokenization | NFC + KS_CHAR_MAP normalization |
Evaluation Metrics
Computed on 16,209 truly-unseen held-out test segments from KS-LIT-3M:
| Metric | Value |
|---|---|
| Composite Quality Score (CQS) | 0.7252 |
| Fertility (↓ better) | 1.2092 |
| Diacritic Preservation Score (↑) | 0.9931 |
| Morphological Boundary Alignment (↑) | 0.1494 |
| Out-of-Vocabulary rate (↓) | 0.0000 |
| Reconstruction (char-level, ↑) | 1.0000 |
See the paper for full evaluation methodology and the linguist-verified gold morpheme reference.
Recommended Use
Primary use case: GPT-style autoregressive generation, machine translation
16K vocab — uses </w> end-of-word marker.
Companion Repositories
The other four KashTok tokenizers are also available for direct comparison:
- Kashmiri_Char_Tokenizer
- Kashmiri_Word_Tokenizer
- Kashmiri_WordPiece_Tokenizer
- Kashmiri_BPE_Tokenizer
- Kashmiri_Unigram_Tokenizer
Citation
@article{malik2026kashtok,
title = {KashTok: Tokenizing Kashmiri at Scale with Novel
Diacritic- and Morphology-Aware Metrics},
author = {Malik, Haq Nawaz and Nissar, Nahfid and others},
year = {2026}
}
Linguistic Verification
Every Kashmiri character, diacritic, and morpheme split used in the evaluation of this tokenizer was confirmed by a native-Kashmiri-speaker linguistic review (40 consonants, 7 vowels, 11 diacritics, 26 gold morpheme splits).
- Library
- transformers
- License
- apache-2.0
- Languages
- ks, kas
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026