tamil-tokenizer-wiki

Tamil tokenizer

BPE for Tamil, trained on cleaned Tamil Wikipedia and an IndicCorpV2 slice. The base vocabulary is 1,135 Tamil orthographic clusters rather than raw bytes, so Grantha syllables and conjuncts cost one token each instead of being reassembled from merges.

Numbers

Held-out splits, not trained on:

Bits-per-byte, cleaned wiki held-out split under Qwen2.5-0.5B, via scripts/colab_bpb.py:

Llama 3 and cl100k_base land 0.007% apart on tokens emitted. Both are byte-fallback BPE with no Tamil vocabulary, so both collapse to about one token per UTF-8 byte. Which byte-level BPE you pick makes no difference; having Tamil vocabulary does.

Measured on the 120K export before the dummy-prefix change; the models published here are slightly better still by bytes/token (see the table above). The gap is under 1% either way.

Files

| tamil_tokenizer_wiki.json | Trainer format: {vocab, merges, metadata}. Load with this repo's Rust encoder. | | tamil_tokenizer_wiki_hf/tokenizer.json | Hugging Face tokenizers format, loadable from Python. | | tamil_tokenizer_wiki_hf/tokenizer_config.json | AutoTokenizer settings and special-token names. | | tamil_tokenizer_wiki_hf/special_tokens_map.json | bos, eos, pad and unk. | | tamil_tokenizer_wiki_120k_mixed.json | Trainer format: {vocab, merges, metadata}. Load with this repo's Rust encoder. | | tamil_tokenizer_wiki_120k_mixed_hf/tokenizer.json | Hugging Face tokenizers format, loadable from Python. | | tamil_tokenizer_wiki_120k_mixed_hf/tokenizer_config.json | AutoTokenizer settings and special-token names. | | tamil_tokenizer_wiki_120k_mixed_hf/special_tokens_map.json | bos, eos, pad and unk. | | tokenizer.json, tokenizer_config.json, special_tokens_map.json | The default 120K model at the repository root, so AutoTokenizer.from_pretrained loads it without a subfolder. |

Use

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("barryallen16/tamil-tokenizer-wiki")
tok.encode("வணக்கம் இது தமிழ்").input_ids

The 120K mixed model sits at the repository root. The 30K wiki model is in tamil_tokenizer_wiki_hf/:

tok = AutoTokenizer.from_pretrained(
    "barryallen16/tamil-tokenizer-wiki", subfolder="tamil_tokenizer_wiki_hf"
)

Without transformers:

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tamil_tokenizer_wiki_hf/tokenizer.json")
tok.encode("வணக்கம் இது தமிழ்").ids

This tokenizer adds no BOS or EOS. tokenizer.bos_token_id and tokenizer.eos_token_id are available for a training loop that wants them.

The bare .json files at the top level are the trainer's own format ({vocab, merges, metadata}) and only load through the Rust encoder. Convert with:

cargo run --release --bin export_hf -- tamil_tokenizer_wiki.json hf_export/out

Training

Both models are v4: train on a copy of the corpus with one space prepended to every line (SentencePiece's add_dummy_prefix). A v3 model differs only in that preprocessing step, so the two published artifacts are comparable.

python scripts/prefix_ab.py data/wiki/train.txt data/wiki/test.txt 30000   # then take dummy.json
python scripts/prep_corpus.py --mix data/mixed 200 wiki+ic+transcripts
python scripts/prefix_ab.py data/mixed/train.txt data/wiki/test.txt 120000  # then take dummy.json

Export fidelity

tokenizer.json round-trips losslessly: 100.0% decode fidelity on both held-out splits, checked before every publish. It reproduces 100.0% of the Rust encoder's token IDs, and the gap is in the encoder.

No merges are lost in the export; all of them are present and in rank order. HF's BPE model starts from single characters and merges every occurrence of the best-ranked pair per pass. The Rust encoder starts from the 1,135 base clusters and merges one occurrence per pass. Reimplementing HF's algorithm over this merge list reproduces its IDs exactly, so the two engines converge once the encoder takes those semantics.

Token count differs by about 1.7% against the Rust encoder. If you need exact ID parity, use the Rust encoder.