Downloads · 30 days
0
balochi-tokenizer
shah-bakhsh/balochi-tokenizer
balochi-tokenizer is a machine learning model from shah-bakhsh. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is a research-grade subword tokenizer for Balochi, a low-resource language spoken by an estimated 15 million people with no prior publicly available tokenizer. It was selected automatically out of 20 candidate to…
Access
Public
Updated Aug 4, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json3.6 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 4, 2026
- Updated
- Aug 4, 2026
- SHA
- 46629ba3
Model card
From the Hugging Face model README
Balochi Tokenizer (hf_bpe, vocab=48000)
Project Overview
This is a research-grade subword tokenizer for Balochi, a low-resource language spoken by an estimated 15 million people with no prior publicly available tokenizer. It was selected automatically out of 20 candidate tokenizers (4 algorithms x 5 vocabulary sizes) using a 10-metric weighted benchmark.
Dataset Description
- Training corpus: 96821 cleaned Balochi sentences (1822326 words, 150127 unique word types)
- Evaluation corpus: pre-tokenized / CoNLL-U style gold-token corpus (28620 sentences) used for morphological boundary agreement
Training Methodology
Trained with hf_bpe at vocabulary size 48000 on identically
cleaned/normalized text (Unicode NFC normalization, deduplication, whitespace
and punctuation normalization) shared across all candidate tokenizers for a
fair comparison.
Evaluation Methodology
16 metrics were computed for every one of the 20 candidate tokenizers:
vocabulary coverage, avg tokens/word, avg tokens/sentence, compression ratio,
vocabulary efficiency, segmentation consistency, UNK rate, tokenization
speed, memory usage, training time, inference latency, model size, token
length distribution, subword frequency, character coverage, and
morphological boundary agreement (vs. CoNLL-U gold tokens). Metrics were
min-max normalized and combined via fixed, pre-declared weights (see
evaluation.csv for full per-tokenizer scores).
Metrics (this tokenizer)
| Metric | Value |
|---|---|
| algorithm | hf_bpe |
| vocab_size | 48000 |
| training_time_sec | 3.0427 |
| model_size_mb | 3.4769 |
| vocab_coverage_pct | 88.8026 |
| avg_tokens_per_word | 1.1703 |
| avg_tokens_per_sentence | 20.9368 |
| compression_ratio_chars_per_token | 4.0725 |
| unk_token_rate_pct | 0.0000 |
| segmentation_consistency_pct | 100.0000 |
| tokenization_speed_sent_per_sec | 21913.1042 |
| inference_latency_ms_per_sentence | 0.0452 |
| rare_word_avg_subwords | 1.9155 |
| character_coverage_pct | 100.0000 |
| morphological_boundary_agreement_pct | 26.9532 |
| vocabulary_efficiency | 1.8501 |
| memory_delta_mb | 5.9023 |
Limitations
- Trained on 96821 sentences; larger corpora may shift optimal vocabulary size.
- Morphological boundary agreement is approximate (offset-based), not a full morpheme-alignment evaluation.
- Domain coverage reflects the source corpus and may not generalize to all registers of Balochi (e.g. heavy code-switching with Urdu/English).
Intended Uses
Subword tokenization for Balochi language modeling, machine translation, OCR/ASR text post-processing, and other downstream NLP tasks requiring a Balochi-aware vocabulary.
License
Apache 2.0
Citation
If you use this tokenizer, please cite the associated GitHub repository: https://github.com/shah-bakhsh/BalTokenizer.git
- License
- apache-2.0
- Languages
- bal
- Created
- Aug 4, 2026
- Updated
- Aug 4, 2026