timemeshin-otm-tokenizer

TimeMeshin-OTM: Universal Spatio-Temporal & Causal Frame Tokenizer

Author: Chandramouli (@Changmaulee) | License: Apache 2.0
GitHub Repository: Changmaulee/timemeshin-otm-tokenizer

---

📌 Executive Summary

Modern Large Language Models (LLMs) often face the "Indic & Multilingual Token Tax": conventional statistical tokenizers (BPE, SentencePiece, WordPiece) sever non-Latin scripts, vowel modifiers (matras), and conjunct consonants into 5–9 raw byte tokens per word. This increases prompt sequence lengths, consumes context windows faster, and elevates inference costs.

While prior pioneering efforts (such as Timegravity and large-vocabulary models) addressed this by expanding the vocabulary to 250k+ tokens, TimeMeshin-OTM (Ordered Transition Mesh) explores a complementary, parameter-efficient structural approach:

• Indivisible Akshara & Glyph Bounds: Enforces phonetic syllable boundaries for Indic/Dravidian scripts and atomic character blocks for CJK, preventing sub-character byte shredding.
• Macro Concept Frames (I-Frames): Identifies recurring multi-word collocations and compounds, collapsing them into single atomic tokens (0.36 – 0.69 tokens/word).
• Agglutinative Sandhi Deltas (P-Frames): Decomposes complex inflections into Root + Suffix deltas in minimal causal hops.
• Ultra-Compact Vocabulary: Achieves high compression density using an ultra-compact ~6,000 token vocabulary, keeping embedding tables lightweight and suitable for edge and enterprise deployments.

---

📊 Comprehensive Empirical Benchmarks

Benchmark 1: Out-of-Distribution (OOD) Zero Data-Leakage Evaluation

Trained strictly on Science/Technology; tested on completely novel domains (Courts, Legal Judgments, Agriculture, Literature, News).

---

⚖️ Architectural Comparison: Structural Frames vs. Vocabulary Expansion

---

📜 License & Attribution

This standalone tokenizer is open-sourced under the Apache License 2.0. You are free to use, modify, distribute, and embed this tokenizer in commercial applications, academic research, and foundation model pre-training.

@software{chandramouli2026timemeshin_otm,
  author = {Chandramouli},
  title = {TimeMeshin-OTM: Universal Spatio-Temporal and Causal Frame Tokenizer},
  year = {2026},
  publisher = {GitHub and Hugging Face},
  url = {https://github.com/Changmaulee/timemeshin-otm-tokenizer}
}