km-improved-22k-v4

🇰🇭 KM Improved 22K V4 Tokenizer

The KM Improved 22K V4 is an upgraded version of the KM Improved 22K tokenizer.
It refines the Khmer subword segmentation process for higher morphological accuracy,
improved reconstruction, and better efficiency on complex Khmer text — including
literature, religious content, and formal writing.

---

🧠 Model Details

Model Description

• Developer: Sok Meas (@Msok99)
• Model Type: SentencePiece Unigram
• Language: Khmer
• License: MIT
• Base Model: Msok99/km-improved-22k
• Vocabulary Size: 22,000
• Objective: Balance between linguistic segmentation quality and efficiency

Model Sources

• Repository: https://huggingface.co/Msok99/km-improved-22k-v4

---

📊 Performance Summary

---

🧩 Uses

Direct Use

• Khmer text tokenization for transformer training or fine-tuning
• Preprocessing for summarization, translation, and QA tasks
• Suitable for large corpus preprocessing (efficient and clean)

Downstream Use

• Chatbots and LLM fine-tuning in Khmer
• Cultural, literary, and religious document understanding

Out-of-Scope Use

• Tokenizing English-only or code-switched text (use merged version instead)
• Tasks requiring semantic embeddings directly (this is a tokenizer)

---

⚖️ Bias and Limitations

• Limited to modern standard Khmer writing conventions
• Slang, abbreviations, or romanized Khmer may not tokenize optimally

Recommendations

For mixed-language text (Khmer + English), use Msok99/lfm2-khmer-merged-18k
to ensure balanced handling of both alphabets.

---

🚀 How to Get Started

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Msok99/km-improved-22k-v4")

text = "ព្រះសង្ឃចេញធ្វើសូត្រមុនព្រះអាទិត្យរះ។"
tokens = tokenizer.tokenize(text)
print(tokens)
print(tokenizer.decode(tokenizer.encode(text)))