lma_phase1-tokenizer

Phase 1 Tokenizers — Hindi and Nepali

Two independent SentencePiece tokenizers trained from scratch for a pair of monolingual ~25M-parameter decoder-only language models. They share no training text, no vocabulary and no merges — the separation is what makes a Hindi/Nepali comparison meaningful.

Chosen models

Each was selected from a sweep of BPE and Unigram × 16,000 and 24,000, then retrained on the complete cleaned corpus. tokenizer_sweep.json in each language folder holds the full comparison.

Why these vocabulary sizes

Fertility improves monotonically with vocabulary, so it cannot be the only criterion. At d_model=384 the embedding table costs vocab × 384 out of a ~25M budget, and one decoder layer costs ~12 × 384² = 1.77M:

A vocabulary much above ~32k spends the model on its own embedding table.

UNK rate is zero by construction

byte_fallback=True: any character the vocabulary cannot cover decomposes into UTF-8 byte tokens (ids 3–258). Nothing is ever discarded.

⚠️ Input must be normalized first

These were trained on text normalized by the project's common/normalize.py, whose behaviour is language-specific: chandrabindu (ँ) folds to anusvara (ं) for Hindi but not for Nepali, where the nasalization is phonemic.

Feeding un-normalized text to the Hindi model triggers byte fallback and inflates the token count. Always normalize with the matching language setting before encoding.

Usage

import sentencepiece as spm
from huggingface_hub import hf_hub_download

path = hf_hub_download("Prateek-Tiwari10/lma_phase1-tokenizer", "hindi/hindi_final.model")
sp = spm.SentencePieceProcessor(model_file=path)
print(sp.encode("प्रधानमंत्री ने कहा कि सरकार काम कर रही है।", out_type=str))

Files

hindi/   hindi_final.model · hindi_final.vocab
         tokenizer_final.json · tokenizer_sweep.json · sweep_models/
nepali/  nepali_final.model · nepali_final.vocab
         tokenizer_final.json · tokenizer_sweep.json · sweep_models/