TR-HASH-Tokenizer-32K

TR-HASH Tokenizer 32K

The 32,000-token ByteLevel BPE tokenizer used by the TR-HASH language-model research line and the 200B-token pretraining mixture.

Properties

• Vocabulary size: 32,000
• Model: ByteLevel BPE
• Normalization: Unicode NFC
• EOS: </s> (ID 0)
• PAD: <pad> (ID 1)
• BOS: <s> (ID 2)
• UNK: <unk> (ID 3)
• Declared model context: 2,048 tokens

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "AETHORIA-AI/TR-HASH-Tokenizer-32K"
)
encoded = tokenizer("Token routing starts with tokenization.")
decoded = tokenizer.decode(encoded["input_ids"])

This repository contains the tokenizer only. It does not define a chat template or a model architecture.

Comparison with GPT r50k_base

r50k_base is used here as the approximately 50K-token GPT reference. The comparison uses transformers without added special tokens for TR-HASH and tiktoken for r50k_base.

TR-HASH uses a 36.3% smaller vocabulary. On the fixed 1,392-character suite included in benchmark_r50k.py, it produces 5.9% more tokens than r50k_base. At hidden size 1,024, the smaller vocabulary removes 18,695,168 embedding parameters, or about 37.4 MB in BF16/FP16, when input and output embeddings are tied.

This small suite covers English prose, technical text, code/JSON, mathematics, and French. It is an illustrative, reproducible check rather than a claim about every corpus. Compression should be measured again on the intended training or deployment distribution.

pip install transformers tiktoken
python benchmark_r50k.py

Associated dataset

AETHORIA-AI/data-32k-200b-tokens