QT-VII-Tokenizer-Family

QT Generation VII Tokenizer Family: Ensemble, Overture & Prelude

Model Lineage: This release supersedes the QT.VI generation. For legacy 64K conversational prose tokenizers, see qt-VI.3.5-64k.

We are releasing the Generation VII tokenizer family: a tiered suite of tokenizers built to address the fundamental realities of modern language model training.

This is a tokenizer-only repository. It does not include model weights; each tokenizer variant is stored in its own subfolder and can be loaded with AutoTokenizer.from_pretrained(..., subfolder=...).

---

Repository Layout

---

The Zero-Sum Boundary of Universal Tokenization

Developing the Generation VII tokenizer family revealed a strict mathematical limit inherent to Byte-Pair Encoding (BPE): vocabulary capacity is a zero-sum macroeconomic space. When simultaneously supporting 204 global languages, dense programming syntax, and scientific mathematics, we reach a hard boundary where data quality can no longer compensate for spatial scarcity.

A tokenizer's vocabulary size represents a fixed parameter budget. Every slot is fiercely contested. Allocating a dedicated subword to prevent a catastrophic byte-fallback in Amharic, Georgian, or Odia inherently consumes a slot that could have been used to compress a deep Python indentation block or a complex LaTeX formula. Once the training corpus is perfectly balanced and the merging algorithms are fully optimised, universal tokenisation ceases to be a data problem and becomes a purely mathematical trade-off.

Consequently, there is no single "perfect" tokenizer. The decision between our Ensemble (129K), Overture (96K), and Prelude (64K) builds represents a deliberate selection of trade-offs. It forces a choice between minimising the model's static embedding memory footprint and maximising cross-domain sequence compression. We believe in being intellectually honest with the developer community about where our artifacts win and where you are better off using something else.

---

Tokenizer Domain Competency Matrix

Based on our rigorous internal benchmarking against frontier commercial tokenizers, here is how the landscape of tokenization currently stands:

---

The Tokenizer Selection Guide

Because tokenizer capacity is a zero-sum space, no single artifact is perfect for every use case. We recommend selecting your tokenizer based on your model size and linguistic target:

• QT.VII.1.8.Plus (Ensemble — 129K): The definitive choice for Frontier Models (8B–70B+) requiring universal language support. Use this if your model must support all 204 FLORES languages with zero script collapses whilst maintaining top-tier Python and programming compression. It leads the QT family in Human Language compression and statistically ties DeepSeek V4 on Python code.

• QT.VII.1.8.Mini (Overture — 96K): The "Goldilocks" zone for Mid-Tier SLMs (1B–8B). If Code is required, we recommend Overture for 90% of SLM use-cases. It slashes embedding VRAM requirements by 25.7% compared to 129K models, keeping sequence expansion for code under 1.5% whilst maintaining complete multilingual health.

• QT.VII.1.8.Micro (Prelude — 64K): The strict requirement for Micro-Models (≤1B). Doing the maths on this, an untied 129K vocabulary eats up over 50% of the parameter budget of a 1B model. Prelude slashes this bloat whilst keeping code compression remarkably tight for agentic and terminal use — improving pure code density by over 19% compared to our legacy 64K build.

• DeepSeek V4-Flash (129K): Highly recommended for High-Resource / Main Language Models. If your project strictly targets major global languages (English, Chinese, Arabic, Russian) and code, but does not need to support regional scripts like Amharic or Odia (which it drops), this tokenizer is exceptionally well-balanced.

• Meta Llama 3.2 (128K): The optimal choice for Western / English-Centric Code Projects. Llama 3 heavily optimises for Western code formats but sacrifices regional and non-Latin script health to do so, resulting in severe byte-fallbacks in Asian and African languages.

• QT.VI.3 (64K): If your project is strictly for a multi-lingual conversational style assistant rather than a coding assistant for terminal use, we advise using our legacy VI.3 build. It retains the highest human prose text density at the 64K tier.

---

Performance & Benchmark Artifacts

We’ve run the maths across three primary evaluation suites to demonstrate these trade-offs visually and quantitatively.

1. TokenizerBench Multi-Domain Summary

This highlights cross-domain parity. Ensemble and Overture remain flat and highly compressed across Bash, Code, Human Languages, and Sci Formulas.

Raw Data — TokenizerBench Suite

Higher Chr/Tok = better compression. Bold = category winner.

---

2. Python Code Compression (Parquet Suite)

Notice the generational leap: Prelude (64K) reduces sequence lengths by over 19% compared to the older VI.3 (64K) on pure Python code. Ensemble (129K) statistically ties DeepSeek V4 while delivering vastly superior regional script coverage.

Raw Data — Python Parquet Benchmark

Corpus: 164 samples | 3,689 lines | 103,724 characters

Lower Total Tokens = better compression. Bold = best-in-class.

---

3. FLORES-200 Regional Script Health

The visual proof of the zero-sum boundary. Whilst all models perform well on European/Global scripts, commercial baselines (Llama 3, DeepSeek) experience massive exponential spikes on regional scripts (Amharic, Georgian, Armenian, Kannada) due to zero-coverage byte-fallback. The QT builds maintain complete structural health.

Raw Data — FLORES-200 Regional Script Groups

The full per-language breakdown is extensive; here are the critical Regional Script results where byte-fallback spikes occur:

Lower token count = better compression (fewer byte-fallbacks). Bold = best among QT.VII family.

<details> <summary><b>Click to expand: Full FLORES-200 per-language token counts</b></summary>

── European & Global ──
English    (eng_Latn): Plus 28,484 | Mini 29,756 | Micro 31,516 | VI.3 35,362 | DeepSeek 27,046 | Llama 27,177
Russian    (rus_Cyrl): Plus 44,098 | Mini 46,962 | Micro 51,445 | VI.3 51,312 | DeepSeek 42,791 | Llama 43,922
German     (deu_Latn): Plus 42,067 | Mini 44,374 | Micro 47,818 | VI.3 44,946 | DeepSeek 40,980 | Llama 43,006
Spanish    (spa_Latn): Plus 41,918 | Mini 44,080 | Micro 47,258 | VI.3 42,424 | DeepSeek 40,393 | Llama 41,846
Greek      (ell_Grek): Plus 68,102 | Mini 73,064 | Micro 79,370 | VI.3 63,873 | DeepSeek 72,490 | Llama 60,655

── Asian Scripts ──
Arabic     (arb_Arab): Plus 41,509 | Mini 44,010 | Micro 48,000 | VI.3 48,180 | DeepSeek 43,707 | Llama 44,912
Hindi      (hin_Deva): Plus 66,408 | Mini 68,023 | Micro 70,870 | VI.3 57,414 | DeepSeek 75,583 | Llama 68,629
Chinese    (zho_Hans): Plus 35,005 | Mini 36,738 | Micro 40,038 | VI.3 38,449 | DeepSeek 25,396 | Llama 34,845
Japanese   (jpn_Jpan): Plus 38,214 | Mini 40,206 | Micro 44,275 | VI.3 37,870 | DeepSeek 38,442 | Llama 41,055
Korean     (kor_Hang): Plus 53,393 | Mini 55,746 | Micro 59,572 | VI.3 40,496 | DeepSeek 44,429 | Llama 40,447

── Rescued Regional ──
Amharic    (amh_Ethi): Plus 54,354 | Mini 58,352 | Micro 63,807 | VI.3 66,775 | DeepSeek 162,518 | Llama 207,122
Georgian   (kat_Geor): Plus 61,931 | Mini 66,862 | Micro 73,878 | VI.3 62,658 | DeepSeek 105,839 | Llama 265,069
Armenian   (hye_Armn): Plus 58,619 | Mini 63,643 | Micro 71,264 | VI.3 54,402 | DeepSeek 81,479 | Llama 235,282
Kannada    (kan_Knda): Plus 80,071 | Mini 83,594 | Micro 88,881 | VI.3 77,639 | DeepSeek 99,659 | Llama 240,682
Lao        (lao_Laoo): Plus 79,756 | Mini 83,564 | Micro 89,856 | VI.3 82,334 | DeepSeek 215,615 | Llama 238,887

</details>

---

Implementation & Usage

You can load these tokenizers directly from this repository via the transformers or tokenizers libraries in Python:

Note: trust_remote_code=True is not required. The tokenizer pipeline, including the code/math-oriented pre-tokenization rules, is serialized in each tokenizer.json.

from transformers import AutoTokenizer

repo_id = "JamesQuartz/QT-VII-Tokenizer-Family"  # Replace with the final Hub repo name if different.

# 1. Load Ensemble (129K) for Frontier / Trillion-parameter Models
tokenizer_ensemble = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder="qt_VII_Plus_129k",
)

# 2. Load Overture (96K) for 3B-8B Code/General Models
tokenizer_overture = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder="qt_VII_Mini_96k",
)

# 3. Load Prelude (64K) for sub-1B SLMs and Agentic Tools
tokenizer_prelude = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder="qt_VII_Micro_64k",
)

For direct tokenizers usage from a local clone:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("qt_VII_Plus_129k/tokenizer.json")

---

Citation

If you use any of the QT Generation VII tokenizers in your research or project, please cite:

@software{quartz_qtvii_2026,
  author = {Downey, James},
  title = {QT Generation VII Tokenizer Family: Ensemble, Overture \& Prelude},
  year = {2026},
  url = {https://huggingface.co/JamesQuartz/QT-VII-Tokenizer-Family}
}

---

Acknowledgements

We would like to extend our gratitude to the creators of the rigorous testing materials that made this evaluation possible:

• KiteFishAI for their comprehensive TokenizerBench suite, which provided the foundational multi-domain evaluations for Bash, Programming Languages, Human Languages, and Scientific Formulas.
• Meta AI (NLLB Team) for the FLORES-200 dataset, which serves as the gold standard for global, multi-lingual script health and allowed us to stress-test regional byte-fallback boundaries across 200+ languages.
• The Open-Source Community (including OpenAI and Google) for the underlying Python coding datasets (HumanEval/MBPP formatted parquet materials) used to push our Python Code Compression benchmarks to the limit.