sentinel-universal-tokenizer

🦴 Sentinel Universal Tokenizer (SUT)

One theorem. Every modality. One vocabulary.

The Sentinel Universal Tokenizer is a multimodal tokenizer that handles text, images, audio, and video in a unified 61,440-token vocabulary, grounded in the Sentinel Manifold mathematics.

🎮 Try it live → Interactive Demo

🧬 Mathematical Foundation

Built on the Gradient Axiom from the Sentinel Manifold:

F(z) = Σ_{n=1}^∞ z^n / n^n    (Sophomore's Dream, Bernoulli 1697)

lim_{z→∞} F'(z)/F(z) = 1/e ≈ 0.367879441171442

📊 Benchmark Results

Deep Benchmark (30 test cases × 4 tokenizers)

Tested across 21 languages + 3 programming languages + math/LaTeX + 7 edge cases:

🏆 Key Result: Vocabulary Efficiency

Sentinel-SUT achieves 3.2× better compression per vocabulary token than Gemma and 2.2× better than Qwen2. Each token does more work — critical for memory-constrained multimodal models.

Per-Language Performance

🏗️ Architecture

┌────────────────────────────────────────────────────────┐
│  SENTINEL UNIVERSAL TOKENIZER (61,440 tokens)          │
│                                                         │
│  [0-32]          → 33 Special / Control tokens         │
│  [33-32,767]     → 32,735 ByteLevel BPE text tokens   │
│  [32,768-49,151] → 16,384 Image codebook tokens       │
│  [49,152-57,343] → 8,192 Audio codebook tokens        │
│  [57,344-61,439] → 4,096 Video codebook tokens        │
│                                                         │
│  Allocation follows 1/e Gradient Axiom                 │
└────────────────────────────────────────────────────────┘

Special Tokens

Codebook Tokens

• 🖼️ Image: <img_0> – <img_16383> (IDs 32,768–49,151) — VQGAN, Cosmos-DI, FSQ
• 🔊 Audio: <aud_0> – <aud_8191> (IDs 49,152–57,343) — EnCodec, SoundStream
• 🎬 Video: <vid_0> – <vid_4095> (IDs 57,344–61,439) — Cosmos-DV

🚀 Quick Start

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("5dimension/sentinel-universal-tokenizer")

# Text
text = "The Sentinel Manifold: F(z) = Σ zⁿ/nⁿ"
tokens = tokenizer.encode(text)
print(f"{len(tokens)} tokens, decoded: {tokenizer.decode(tokens)}")

# Multimodal (text + image VQ indices)
text = "<image_start> <img_42> <img_1337> <image_end> Describe this."
tokens = tokenizer.encode(text)
for tid in tokens:
    if 32768 <= tid < 49152:
        print(f"  IMAGE codebook[{tid - 32768}]")

# Chat
chat = "<system>Multimodal AI</system><user>What is 1/e?</user><assistant>"
tokens = tokenizer.encode(chat, add_special_tokens=False)

🔬 Innovations

1. 1/e Vocabulary Allocation — Gradient Axiom ratio allocates tokens across modalities
2. ByteLevel BPE — Handles all Unicode, no UNK possible, NFKC normalized
3. 20-language training — EN, FR, DE, ES, ZH, JA, AR, RU, KO, HI, PT, IT, NL, PL, VI, TH, TR, UK, SV + code + math
4. Native Multimodal Routing — Single integer comparison determines modality
5. Sentinel Manifold Integration — Special tokens for manifold-aware computation

📦 Training

🔗 Links

• 🎮 Interactive Demo
• 🦴 Sentinel Manifold Framework
• 📜 Training scripts included in repo

📚 Citation

@misc{abdel-aal2026sentinel-tokenizer,
  title={Sentinel Universal Tokenizer: Multimodal Tokenizer Grounded in the Gradient Axiom},
  author={Abdel-Aal, Romain},
  year={2026},
  url={https://huggingface.co/5dimension/sentinel-universal-tokenizer}
}

---

Built by: Romain Abdel-Aal (ASI The Sentinel V5.2 Bone-Core) · MIT License · 🦴