geez-en-shared-tokenizer

geez-en-shared-tokenizer

A shared SentencePiece Unigram tokenizer (vocab size 120,000, byte_fallback=true) covering Amharic, Tigrinya, Tigre, Classical Ge'ez, and English. Built as a continuation of "MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages" (Teklehaymanot, Fazlija, Nejdl — L3S Research Center).

Usage

import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
print(sp.encode("አልሰበሩም", out_type=str))

Training data — provenance and licenses

Trained on a temperature-sampled (alpha=0.3) mix of cleaned text from these sources. Licenses are mixed and one local source is unspecified — review before treating this as uniformly open-licensed for downstream redistribution of text generated by decoding, though the vocabulary/subword-piece list itself does not reproduce the training text.

Full per-source breakdown: see SOURCES.md in the GeezTokenizer project this was built from.

Limitations

• Bilin (byn) has no dedicated training data — confirmed absent from every broad-coverage source checked (HF Hub, GlotCC-V1, FineTranslations, biblenlp-corpus-mmteb, GitHub). byte_fallback=true lets this tokenizer still encode Bilin text losslessly, but with no dedicated subword coverage.
• Ge'ez is extremely low-resource — only 2,107 sentence pairs of real training text exist anywhere that could be found; its ~1,800 unique sentences were each repeated ~478x during temperature-sampled balancing to give it fair representation in the shared vocabulary. This is a documented, independently-reconfirmed data-scarcity constraint (matches the MoVoC paper's own reported limitation for the same two languages), not a search gap.
• Tigre text sourced from CommonCrawl-derived corpora (GlotCC, FineTranslations) is filtered by language-ID confidence (≥0.5) but should still be treated as secondary to the dedicated BeitTigreAI corpus — automatic language-ID is known to confuse Tigre with the much more common Tigrinya.

Validation

0% UNK rate across all 5 languages; fertility 1.2–1.4 tokens/word. Full report and methodology in the source project's 04_validation/report.md.

Installation

pip install tokenizers huggingface_hub

Tokenizer configuration

Character coverage, sampling temperature, normalisation rules, and the per-language corpus sizes used for temperature sampling are Not documented.

Citation

If you use this artifact, please cite:

@inproceedings{teklehaymanot2025movoc,
  title     = {MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages},
  author    = {Teklehaymanot, Hailay Kidu and Fazlija, Dren and Nejdl, Wolfgang},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year      = {2025},
  url       = {https://arxiv.org/abs/2509.08812}
}