pt-es-hi-tokenizer

pt-es-hi-tokenizer

A shared multilingual byte-level BPE tokenizer (GPT-2/Llama-3 style) for Portuguese, Spanish, and Hindi — one vocabulary, not per-language, intended for a single multilingual pre-training run rather than three separate models.

• Vocab size: 32,000, chosen via the Kneedle algorithm (point of maximum curvature on the compression-vs-vocab-size curve) over a candidate sweep of 8k/16k/32k/50k/65,536/100k, rather than a hand-picked threshold.
• Byte-level pre-tokenization: every input string is representable with no OOV/<unk> fallback, which matters since the training corpus mixes Devanagari (Hindi) and Latin (Portuguese/Spanish) scripts.
• Special tokens follow the Llama-3 chat convention (<|begin_of_text|>, <|end_of_text|>, <|pad|>, <|start_header_id|>/<|end_header_id|>/<|eot_id|>), so the tokenizer is chat-fine-tuning-ready without resizing the embedding table later.

Training data

Trained on a MinHash-deduplicated corpus of pt-fineweb2, es-fineweb2, hi-fineweb2, and hi-sangraha (ai4bharat/sangraha's "verified" split), deduplicated per-language with datatrove (hi's two sources deduplicated against each other, not just internally).

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("andre15silva/pt-es-hi-tokenizer")
tok.encode("Olá, ¿cómo estás? नमस्ते")

Evaluation

Fertility (tokens/word, lower is better) and compression (bytes/token, higher is better) on a disjoint held-out sample:

Hindi's fertility is notably higher than pt/es — consistent across every scale this was measured at (pilot, pre-dedup, and this final run), pointing to Hindi being comparatively data-starved in the training corpus rather than a tokenizer-vocab problem.