qt-VI.1.3-32k

QT VI.1.3 UltraLingo (32K, Prelude)

A 32,000-token multilingual byte-level tokenizer covering the full FLORES-200 set (204 languages, 27+ scripts) with zero coverage failures. VI.1.3 reads as 6.1.3 in Quartz notation (the leading Roman V is the numeral 5, so the VI generation is 6). This is the Prelude tier: a 32K tokenizer tuned to keep English and the major languages competitive with industry 32K tokenizers while covering the entire long tail losslessly. The planned Overture tier (64K) is intended to raise the low-resource complex scripts without an English trade-off.

Predecessor: JamesQuartz/qt-V.4.6-32k.

TL;DR

At the same 32K budget as Llama 2 and Mistral, QT VI.1.3 is a genuinely better multilingual tokenizer. It covers all 204 FLORES-200 languages losslessly, where Llama 2 and Mistral each fall back to bytes on 34; it beats Llama 2 on 107 of the 170 languages Llama can tokenize at all, and Mistral on 137; and it roughly doubles their efficiency on Arabic, Hindi and Thai while leading on CJK. The cost is a modest, honest tax on English and European text (about 8% behind Llama 2), the price of universal coverage at a fixed vocabulary.

What it is

• Vocabulary: 32,000 tokens.
• Algorithm: two-stage byte-level BPE. Stage 1 learns subwords; Stage 2 learns cross-word "superwords" under a BoundlessBPE sanity rule, so partial-word fragment merges (the "e t" junk class) are impossible by construction.
• Coverage: all 204 FLORES-200 languages, 27+ scripts, no zero-coverage script.
• Base: byte-level. Every byte sequence is representable, so there is no UNK token and decoding is exactly lossless.

Correctness guarantees

• Lossless round-trip on all 204 languages: decode(encode(x)) == x byte-for-byte.
• No UNK emission anywhere.
• No byte-fallback collapse; no zero-coverage script.
• FLORES-200 health gate: 0 correctness failures across all 204 languages.

Head-to-head vs industry 32K tokenizers (FLORES-200 devtest)

Coverage

Llama 2 and Mistral have no dedicated tokens for many scripts (for example CJK, Tibetan, Telugu, Myanmar, Armenian, Lao) and fall back to raw bytes, which the coverage check flags as failures.

Compression, chars per token (higher is better)

QT trails Llama 2 by about 8% on English and the European languages, sits at or above Mistral on German, Italian and Spanish, and is far ahead of both on every non-Latin script. Across the 170 languages Llama 2 can tokenize, QT wins on 107; against Mistral it wins on 137.

vs the predecessor, QT V.4.6 (FLORES-200, chars per token, higher is better)

VI.1.3 fixes the English over-fragmentation that motivated this line. It improves on 98 of 204 languages, is more equitable overall (spread 6.4x vs 7.6x), and lifts English and the major languages sharply; the trade is on CJK and the low-resource complex scripts, which are the 64K Overture target.

English

English was the motivating case for this line. VI.1.3:

• 1.53 tokens per word on running English text, at 3.90 chars per token, which trails Llama 2's 4.24 by about 8%.
• Common long words are single or paired tokens: committee and recommended take 2 tokens each, international and development a single token.

This is competitive with industry 32K tokenizers on English while retaining full multilingual coverage. Closing the final few percent to match Llama 2 exactly is planned for the Overture (64K) tier, where the larger budget is expected to remove the coverage-versus-efficiency trade.

Special tokens

The tokenizer reserves 86 special tokens (IDs 0-85) for downstream model use:

• Core: <|padding|> (0), <|bos|> (1), <|endoftext|> (2, EOS), <|unk|> (3), <|sep|> (4). The <|unk|> token is declared for convention only; the byte-level base means it is never emitted.
• Chat and tools: <|system|>, <|user|>, <|assistant|>, <|tool_call|>, <|tool_result|>, <|thinking|>, <|/thinking|>, <|code|>, <|/code|>.
• Language control: 72 <|lang:xx|> tags (for example <|lang:en|>, <|lang:ja|>, <|lang:ar|>) for optional language steering.

These are reserved slots; no chat template is bundled, so a downstream model can define its own.

Usage

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")

enc = tok.encode("The committee recommended further investigation.")
print(enc.ids)
print(enc.tokens)

# Lossless round-trip across scripts
text = "中华人民共和国 / भारत / ประเทศไทย / العربية"
assert tok.decode(tok.encode(text).ids) == text

With Hugging Face Transformers:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("JamesQuartz/qt-VI.1.3-32k")
tok("The quick brown fox")

Files: tokenizer.json (self-contained), plus vocab.json and merges.txt for compatibility.

Version notation

Quartz uses a leading Roman V as the numeral 5. So V.4.6 reads as 5.4.6, and VI.1.3 reads as 6.1.3: the sixth-generation architecture (VI), first line (.1), third iteration (.3). Never written V6.1.3.

Intended use and limitations

A general-purpose multilingual tokenizer for pretraining and inference across the full FLORES-200 set, tuned so English and major languages stay competitive with industry 32K tokenizers while nothing in the long tail is left uncovered. At 32K it is a strong generalist rather than a per-language optimum; the lowest-resource complex scripts (for example Tibetan) are constrained by both the fixed budget and available training data, and are the target of the 64K Overture tier.

Citation

@misc{qt_vI_1_3_ultralingo,
  title  = {QT VI.1.3 UltraLingo: a coverage-complete 32K multilingual tokenizer},
  author = {Quartz (AENEA Global)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/JamesQuartz/qt-VI.1.3-32k}}
}