qt-v4.1-64k-ultralingo

QT V.4.1 64K UltraLingo — SuperBPE Tokenizer

Quartz Data Infrastructure — quartz.host | AENEA Global — aeneaglobal.com

A 64,000-vocabulary multilingual BPE tokenizer covering 71 languages across 27 scripts, designed for the AENEA Overture model series (500M–2B parameters). Part of the QuartzTokenizer (QT) family.

Key Results

Benchmarked on FLORES-200 (204 languages, 1,012 parallel sentences each):

QT V.4.1 64K achieves lower mean fertility with half the vocabulary, 3.7x better cross-lingual equity, and 22.6% fewer total tokens than Llama 3.

Architecture

QT V.4.1 is a two-stage SuperBPE tokenizer with three innovations over standard BPE:

1. Two-Stage SuperBPE Training

• Stage 1 (57,600 tokens, 90%): Standard BPE with Llama 3 / GPT-4 style whitespace pre-tokenization. Learns subword units — roots, affixes, morphemes, character sequences.
• Stage 2 (6,400 tokens, 10%): SuperBPE — lifts the whitespace boundary constraint, allowing merges to span across word boundaries. Learns high-frequency multi-word superword tokens (e.g., of the, in order to). Sentence boundary protection prevents cross-sentence tokens.

Based on Liu et al., COLM 2025 — "SuperBPE: Space Travel for Language Models" (+4.0% downstream, +8.2% MMLU, −27% inference compute).

2. Script-Aware Pre-Tokenization (Indic Only)

• Virama-aware character segmentation for Indic scripts (Devanagari, Bengali, Tamil, Telugu, Kannada, Malayalam, Gujarati, Gurmukhi, Odia, Sinhala)
• Preserves conjunct consonants by not breaking across virama (halant) marks
• CJK, Thai, Khmer, Myanmar, and Tibetan are left as raw text to enable proper multi-character merge learning

3. Streaming Sharded Training

• Corpus sharded to disk for RAM-bounded training
• Separate sample ratios and minimum frequencies for Stage 1 and Stage 2
• Enables SuperBPE training on consumer hardware (16 GB RAM)

Training Data

Trained on a balanced multilingual corpus (~5 GB target, 0.35 effective sample ratio):

Corpus design follows "The Art of Breaking Words" (arXiv 2508.06533) iterative fertility balancing and "One Tokenizer to Rule Them All" script/family bucket approach.

Per-Script Performance

FLORES-200 benchmark — mean tokens per word (lower is better):

Special Tokens

14 structural tokens + 71 language tags = 86 special tokens total.

Usage

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")

# Encode
encoded = tok.encode("The history of the Roman Empire spans centuries.")
print(encoded.ids)      # Token IDs
print(encoded.tokens)   # Token strings

# Decode
text = tok.decode(encoded.ids)
print(text)

Intended Use

QT V.4.1 64K is designed as the tokenizer for the AENEA Overture model series (500M–2B parameters). It is optimised for:

• Multilingual language modelling across 71 languages
• Cross-lingual transfer with equitable compression across scripts
• Code generation (Python, JavaScript, Java, C/C++, Go, Rust)
• Mathematical and scientific text
• Instruction-following with dedicated chat tokens

Recommended Pairing

Training Configuration

Algorithm:            SuperBPE (two-stage)
Stage 1 vocab:        57,600 (90% — subword with whitespace boundaries)
Stage 2 vocab:        6,400 (10% — superword, no whitespace constraint)
Min frequency:        Stage 1: 2, Stage 2: 50
Sample ratio:         Stage 1: 0.35, Stage 2: 0.08
Pre-tokenization:     Script-aware (Indic virama segmentation)
Training mode:        Streaming sharded (500 MB shards)
Seed:                 42
Training time:        ~111 minutes (RTX 4060, 16 GB RAM)

Limitations

• Tibetan (33.9 TPW) has improved significantly over previous versions but is still high due to the lack of whitespace delimiters. Future versions will increase the Tibetan corpus weight.
• Scripts without whitespace (Thai, Khmer, Tibetan, CJK) inherently require more tokens per word under BPE with whitespace pre-tokenization.
• The tokenizer is trained for tokenization quality, not for any specific downstream task. Model performance depends on the language model trained on top.

References

• Liu et al., COLM 2025 — "SuperBPE: Space Travel for Language Models"
• arXiv 2511.03237 — IndicSuperTokenizer: SOTA fertility on 22 Indic languages
• arXiv 2508.06533 — "The Art of Breaking Words": iterative fertility-driven reweighting
• Tao et al., NeurIPS 2024 — Scaling Laws with Vocabulary
• arXiv 2601.20994 — "The Depth Delusion": width > depth, 32K optimal for sub-500M
• NeurIPS 2025 Workshop — "From Bias to Balance": balanced tokenizer datasets
• Arnett et al. 2025 — Crosslingual Tokenizer Inequities

Citation

@misc{downey2026qt,
  title={QT V.4.1 UltraLingo: A Streaming Script-Aware SuperBPE Tokenizer for Equitable Multilingual Language Modelling},
  author={Downey, James},
  year={2026},
  publisher={AENEA Global Ltd},
  url={https://huggingface.co/JamesQuartz/qt-v4.1-64k-ultralingo}
}

About

Built by James Downey at AENEA Global Ltd (Company No. 16743851, Manchester).

• Quartz — Open-source data pipelines and tokenizers (quartz.host)
• AENEA — Language model laboratory (aenea.app)
• Crassus — Institutional credit intelligence (crassus.info)