sinhala-bpe-32k

sinhala-bpe-32k

A SentencePiece BPE tokenizer for Sinhala (32k vocab, character coverage 0.9999), trained on the deduplicated OSCAR Sinhala corpus.

On Sinhala text it reaches fertility 1.58 (tokens per word) with UNK rate ≈ 0.0007 — 1.3–2× more efficient than 200k–250k multilingual vocabularies (XLM-R 2.04, GPT-4o's o200k 3.26, GPT-4's cl100k 9.80), measured with tokscope. Full comparison: the Sinhala tokenizer study.

Usage

import sentencepiece as spm
from huggingface_hub import hf_hub_download

path = hf_hub_download("RavinduPabasara/sinhala-bpe-32k", "sinhala_bpe_v2.model")
sp = spm.SentencePieceProcessor(model_file=path)
sp.encode_as_pieces("මම සිංහල භාෂාව ඉගෙන ගනිමි.")
# ['▁මම', '▁සිංහල', '▁භාෂාව', '▁ඉගෙන', '▁ගනි', 'මි', '.']

Special ids: pad=0, unk=1, bos=2, eos=3.

Intended uses

• Tokenizer extension / vocabulary adaptation when fine-tuning multilingual models for Sinhala
• Training Sinhala-first language models
• A reference point for tokenizer-efficiency comparisons on Sinhala

Training data

OSCAR Sinhala subset (~100k lines), cleaned with the pipeline in SinhalaCustomTokenizer. OSCAR's own terms apply to the underlying text; this repository distributes only the trained tokenizer model, not the corpus.

Author

Ravindu Pabasara Karunarathna — BSc (Hons) in AI undergraduate, University of Moratuwa, Sri Lanka.