qt-VI.3.5-64k

QT VI.3.5 UltraLingo (64K)

A 64,000-token multilingual byte-level BPE tokenizer covering 204 FLORES-200 languages losslessly. It is the flagship of the Quartz tokenizer line: built to beat the widely used 32K tokenizers on the world's major languages while keeping every other language strongly ahead of them, at a vocabulary size suited to 500M-1B parameter models.

Trained by JamesQuartz. Previous release: qt-VI.1.3-32k.

What it is

• 64,000 tokens, two-stage byte-level BPE (subword Stage 1 + BoundlessBPE sanity-superword Stage 2).
• Byte-level base: no UNK, lossless decode for any input, no script can hit zero coverage.
• 204 / 204 FLORES-200 languages represented with real subword inventories, not byte fallback.
• 86 special tokens at fixed IDs (padding, bos, endoftext, unk, sep, chat/tool/ code markers, and 72 <|lang:xx|> tags).
• Digit grouping, Indic virama-aware segmentation, and dense-script chunking in the pre-tokenizer.

Why 64K, and who it is for

Optimal vocabulary size grows with model size (Tao et al., 2024). 64K is the right tier for 500M-1B parameter models: large enough to give the major languages whole-word coverage and the complex scripts a real subword inventory, without spending capacity that a sub-2B model would rather use on transformer layers. For smaller models the 32K qt-VI.1.3-32k remains the right choice.

Benchmarks

All numbers are from FLORES-200 devtest across 204 languages. The primary metric is bytes per token (BPT) - higher means more text compressed per token - which is well-defined for every script. Compared against Llama 2 and Mistral (both 32K).

Coverage and global compression

Llama 2 and Mistral fail to cover 34 languages, falling back to raw bytes on entire scripts. QT VI.3.5 covers all 204 losslessly and compresses ~41% more per token on average than either.

Major languages (chars per token, higher is better)

QT VI.3.5 beats both Llama 2 and Mistral on 11 of these 13 languages, often by 2-3x on CJK, Arabic and Hindi. It ties Llama 2 on German. English is the one language where a dedicated 32K English/European tokenizer still leads - see Honest limitations.

By script family (bytes per token, higher is better)

Versus the previous flagship

QT VI.3.5 improves compression over the 32K QT VI.1.3 on 198 of 204 languages. The only regression is English (marginal, and by design - see below).

Honest limitations

• English. A 64K tokenizer covering 204 languages cannot out-compress a 32K tokenizer that spends its entire vocabulary on English and a few European neighbours. QT VI.3.5 reaches English parity with the previous 32K flagship (3.84 vs 3.90 chars per token) but trails Llama 2's 4.24 by design. This is a structural property of broad coverage, not a defect: the budget that would push English further is spent giving 200+ other languages real representation. If your use case is English-only, a dedicated English tokenizer will compress English better.
• Deepest low-resource scripts. Languages with very little training data (e.g. Tibetan) improve but remain limited by corpus availability rather than vocabulary size. More data, not more vocabulary, is the fix.
• The tokenizer targets compression and coverage. Downstream model quality depends far more on model scale and training-data mix than on tokenizer choice.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("JamesQuartz/qt-VI.3.5-64k")
ids = tok.encode("The committee recommended further investigation.")
print(tok.decode(ids))

Or with the tokenizers library directly:

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
enc = tok.encode("中华人民共和国成立于一九四九年十月一日。")
print(enc.tokens)

Files

• tokenizer.json - the complete tokenizer (model, pre-tokenizer, decoder, specials).
• vocab.json, merges.txt - vocabulary and merge rules.
• tokenizer_config.json, special_tokens_map.json - HuggingFace config.

License

Apache-2.0.

Citation

@misc{qt-vi35-ultralingo-2026,
  title  = {QT VI.3.5 UltraLingo: A 64K Multilingual Tokenizer for 204 Languages},
  author = {JamesQuartz},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/JamesQuartz/qt-VI.3.5-64k}}
}