tigre-spm-tokenizer

Tigre SentencePiece Tokenizer (16K Vocab)

A standalone SentencePiece (Unigram/BPE) subword tokenizer built and optimized for the Tigre language (ትግሬ) written in Ge'ez script.

Tokenizer Statistics & Verification

Quickstart Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BeitTigreAI/tigre-spm-tokenizer")

text = "ሰላም፡ እሊ ናይ ሂጋ ትግሬ ክታበት ቱ።"

# Encode to tokens & IDs
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("IDs   :", input_ids)

# Lossless Decoding
decoded_text = tokenizer.decode(input_ids, clean_up_tokenization_spaces=False)
print("Decoded:", decoded_text)
assert text == decoded_text

Repository Structure

• tokenizer.model: Native SentencePiece binary model file.
• tokenizer.json: Serialized fast tokenizer representation for Python/Rust environments.
• tokenizer_config.json: Metadata and special token configuration mapping.
• special_tokens_map.json: Explicit <pad>, <s>, </s>, and <unk> assignments.