khmer-sp-8k

khmer-sp-8k

A SentencePiece unigram tokenizer (vocab size 8000) for Khmer with code-switched English, trained on Panhapich/khmer-text-corpus. It is the shared vocabulary for OCR, ASR, and the diffusion decoder in the Khmer multimodal project.

The tokenizer is several files working together, not one

khmer_sp.model was trained on word-segmented text (via khmer-nltk), not raw Khmer — this is what prevents SentencePiece from fusing multiple words into a single vocab token (e.g. ខ្ញុំចង់ = "I"+"want" collapsing into one piece). Glued English/Latin runs (e.g. tryAImodel) are also decomposed via wordninja, guarded by a case-sensitive exception list so domain terms (ACLEDA, Bakong, COVID-19, 5G, ...) aren't mangled. See khmer_segmentation_sentencepiece_approach.md for the full root-cause writeup.

This means khmer_sp.model must never be loaded with a bare SentencePieceProcessor for encode/decode. Always go through khmer_segmentation.KhmerTokenizer, which runs the same segmentation pipeline used at training time:

from huggingface_hub import hf_hub_download
from khmer_segmentation import KhmerTokenizer

sp_model_path  = hf_hub_download("Panhapich/khmer-sp-8k", "khmer_sp.model")
gazetteer_path = hf_hub_download("Panhapich/khmer-sp-8k", "gazetteer.json")
latin_path     = hf_hub_download("Panhapich/khmer-sp-8k", "latin_exceptions.json")

tok = KhmerTokenizer(sp_model_path, gazetteer_path, latin_path)

text = "ខ្ញុំកំពុងប្រើ AI model"
ids = tok.encode(text)
print(ids)
print(tok.decode(ids))

MASK = tok.sp.piece_to_id("<MASK>")
assert MASK not in (-1, tok.sp.unk_id())   # <MASK> must be a real token

Also download khmer_segmentation.py itself (hf_hub_download("Panhapich/khmer-sp-8k", "khmer_segmentation.py")) and import it locally, or vendor it into your own project — it is a required part of the tokenizer, not optional preprocessing.

Files

Special tokens

<MASK> is a SentencePiece user_defined_symbol, so it is one atomic token and must not collapse onto <UNK>. The authoritative IDs are in tokenizer_info.json.

Training details

Licensing

Trained on a corpus derived from third-party datasets; distribution terms follow that corpus. Confirm the upstream licenses and set the license field accordingly — other is a placeholder.