WWHO

Separate before you Compress

<!-- **Remeinium Research** [remeinium.com](https://remeinium.com) | [Paper](https://arxiv.org/abs/...) | [Tokenizer](https://huggingface.co/remeinium/WWHO) | [Dataset](https://huggingface.co/datasets/remeinium/WWHO_Cleaned_30m) --- -->

The Next Architectural Primitive in Tokenization

Large language models remain linguistically blind to Abugida scripts. Byte-Pair Encoding and its descendants routinely shatter complex conjuncts — atomic multi-codepoint grapheme clusters that constitute the fundamental phonetic units of Indic and Southeast Asian writing systems — into meaningless sub-character fragments. The result is degraded reasoning, inflated inference costs, and a systemic “Token Tax” that disproportionately burdens more than one billion speakers.

WWHO (Where-What-How Often) introduces the clean separation of concerns the field has been missing.

By decoupling linguistic structural constraints from statistical compression, WWHO builds a unified meta-vocabulary space:

1. Layer 1 (Where): Code-Switching Router
   A linear $O(N)$ block scanner that evaluates characters in $O(1)$ time to inherently identify script boundaries, routing Latin text to proven frontier tokenizers (like o200k_base) while sending Abugida text for specialized processing.
2. Layer 2 (What): LinguisTrie
   Enforces linguistic integrity by construction: a DFA based syllabifier segments raw Unicode into well-formed syllables with a formal zero-breakage guarantee.
3. Layer 3 (How Often): SGPE & Meta-Vocabulary
   Performs statistical pair merging exclusively over this linguistically sound stream, safely projecting the resulting tokens into a unified, mathematically offset ID space.

Sinhala and Devanagari serve as the high-complexity proofs-of-concept. The same architecture generalizes directly to Tamil, Khmer, Myanmar, and the broader Abugida family.

---

Multi-Script Stratified Benchmarks (122.2M Characters)

We evaluated WWHO against frontier models across a 1.5 million sentence code-switched corpus containing Sinhala, Hindi (Devanagari), and English.

1. Sinhala Efficiency

2. Hindi (Devanagari) Efficiency

3. English

(Note: Because WWHO routes Latin text directly to the native Tiktoken sequence, English performance is mathematically identical. The minor delta in total tokens emerges solely from boundary crossing mechanics.)

4. Overall (Mixed-Script)

• Zero-Breakage Guarantee: Validated through exhaustive testing permutations across all supported Abugida scripts (0 violations).
• Full-corpus reconstruction: 1.5M code-switched sentences encoded and decoded with 0 non-UNK mismatches.
• UNK rate: 0.08 % (restricted strictly to rare compounds without violating structural boundaries).

WWHO radically compresses the context window for Abugida text, effectively ending the Token Tax without penalizing existing state-of-the-art programming and reasoning capabilities.

---

Quick Start with Hugging Face

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("remeinium/SGPE")
text = "ආයුබෝවන් ශ්‍රී ලංකා"

tokens = tokenizer.tokenize(text)
# ['ආයුබෝවන්', ' ශ්‍රී', ' ලංකා']
print(tokenizer.encode(text))

---

Resources

<!-- - **Research Paper**: “The Syllable is the Token: Breaking the Token Tax with SGPE” (Remeinium Research, February 2026) -->

• Pre-trained Tokenizer: Hugging Face
• Cleaned Training Corpus: Hugging Face
• Full Code & Evaluation Harness: GitHub

---

License

Apache License 2.0 — see LICENSE.

---

Citations

<div style="background: #1d1d1d; padding: 1.5rem; border-radius: 8px; border: 1px solid #222; font-family: sans-serif;"> <p style="font-size: 1.1rem; line-height: 1.6; color:#e0e0e0;"> <strong>Title:</strong> Separate Before You Compress: The WWHO Tokenization Architecture<br> <strong>Author:</strong> Kusal Darshana<br> <strong>Year:</strong> 2026<br> <strong>DOI/ArXiv:</strong> <a href="https://arxiv.org/abs/2603.25309" style="color: #e2e4bf; text-decoration: underline;">2603.25309</a> </p> <details style="margin-top: 1rem;"> <summary style="cursor: pointer; color: #d78cee; font-weight: bold;">View BibTeX</summary> <pre style="background: #2d2d2d; color: #e9e9e9; padding: 1rem; border-radius: 6px; overflow-x: auto; font-size: 0.9rem; margin-top: 0.5rem;"> @article{darshana2026separate, title={Separate Before You Compress: The {WWHO} Tokenization Architecture}, author={Darshana, Kusal}, eprint={2603.25309}, eprinttype={arxiv}, year={2026} } </pre> </details> </div>

Remeinium Research | Remeinium AI | Intelligence for a Greater Tomorrow

---