JiRack-Ultra-Tokenizer-256K

Enjoy — We extend the JiRack Models Ecosystem! 🚀

JiRack Utra Tokenizer 256

JiRack Utra Tokenizer - 347 active language editions of Wikipedia

High-performance production-grade Byte-Level BPE tokenizer developed as part of the JiRack Ternary Models ecosystem.

This is the Ultra version designed for maximum quality, better compression vs 128k Pro version, and precision in complex real-world applications.

• JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

Open Robot platform

• Tiangong : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
• Unitree g1 https://a.co/d/0e4A8YVc
• LimX Oli https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
• ubtrobot https://www.ubtrobot.com/en/
• x-humanoid https://www.x-humanoid.com/detail/hskw.html

ROBOS HUMANOID Project

• https://github.com/cmsmanhattan/ROBOS

Robot dataset example

{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: a table in front of me, with a red mug on it. Command: pick up the mug."},{"role":"assistant","content":"reach(x=0.4,y=0.0,z=0.3)"}]}
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and .ROBOTICS"},{"role":"user","content":"Scene: hand near the mug, distance 2 cm. Command: grasp."},{"role":"assistant","content":"grasp(force=60)"}]}
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: mug in hand, stable. Command: lift 20 cm."},{"role":"assistant","content":"lift(z=0.2)"}]}

{
 "messages": [
 {
 "role": "system",
 "content": "You are a robot. Use tags and . ROBOTICS"
 },
 {
 "role": "user",
 "content": "Scene: a table in front of me, with a red mug on it. Command: pick up the mug."
 },
 {
 "role": "assistant",
 "content": "reach(x=0.4,y=0.0,z=0.3)"
 }
]
}

Key Features

• Algorithm: Byte-Level BPE
• Vocabulary Size: 262,144 tokens — excellent balance between precision and efficiency
• Multilingual & Technical Strength: Optimized for English, Russian, code, scientific literature, and technical documentation
• Domain Specialization: Strong performance on programming languages, engineering, robotics, and scientific texts

Special Tokens Support

• Full ChatML dialogue format (<|im_start|>, <|im_end|>)
• FIM (Fill-in-the-Middle) support for code generation
• Rich set of domain routing tokens (__CODING__, __PYTHON__, __ROBOTICS__, __SCIENCE__, etc.)
• Extended robotics and control tokens

CMS Manhattan Service & Support

• Jirack patent guards your technology for competitors
• Redesign Llama , Qwen , Gemma to Ternary model
• Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347
• Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Ultra-Tokenizer-256K")

text = "__CODING__ __PYTHON__ Write a merge sort function in Python."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("Token IDs:", token_ids)

Vocab size: 262144
pad_token_id: 2
eos_token_id: 1

JiRack Pretrain Dataset

https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]

Benchmark for tokens quality .

(venv_ji) root@jirack2:/mnt/nfs_share/JiRackTokenizerPro_256K# python test_v1.py
=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 27953, 522, 60261, 6087, 1810, 576, 6530, 269, 4, 454]

=== Decoding Token by Token ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
27953 -> ' Write'
522 -> ' a'
60261 -> ' merge'
6087 -> ' sort'
1810 -> ' function'
576 -> ' in'
6530 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>


=== Токены (ID) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 33218, 51217, 18533, 70799, 61781, 159246, 2230, 7940, 1078, 1135, 1227, 917, 57473, 269, 4, 454]

=== Декодирование по токенам ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
33218 -> ' Нап'
51217 -> 'иши'
18533 -> ' функ'
70799 -> 'цию'
61781 -> ' сор'
159246 -> 'тиров'
2230 -> 'ки'
7940 -> ' сл'
1078 -> 'ия'
1135 -> 'ни'
1227 -> 'ем'
917 -> ' на'
57473 -> ' python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>


=== Token (ID) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 190845, 6530, 476, 24555, 16049, 57962, 14098, 19575, 26588, 88529, 882, 4, 454]

=== 逐个 Token 解码 ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
190845 -> ' 用'
6530 -> ' Python'
476 -> ' '
24555 -> '写'
16049 -> '一个'
57962 -> '归'
14098 -> '并'
19575 -> '排'
26588 -> '序'
88529 -> '函数'
882 -> '。'
4 -> '<|im_end|>'
454 -> '
'


=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>


=== الرموز (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 35153, 9711, 935, 30958, 199128, 6585, 5673, 1201, 578, 108628, 6087, 264, 6685, 29111, 76156, 2731, 1452, 269, 4, 454]

=== فك الترميز رمزا برمز ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
35153 -> ' اك'
9711 -> 'تب'
935 -> ' د'
30958 -> 'الة'
199128 -> ' فرز'
6585 -> ' بال'
5673 -> 'دم'
1201 -> 'ج'
578 -> ' ('
108628 -> 'merge'
6087 -> ' sort'
264 -> ')'
6685 -> ' بل'
29111 -> 'غة'
76156 -> ' باي'
2731 -> 'ث'
1452 -> 'ون'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>


=== Token (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 118471, 18077, 7384, 237233, 268, 124070, 268, 67699, 32813, 576, 6530, 269, 4, 454]

=== Dekodierung Token für Token ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
118471 -> ' Schre'
18077 -> 'ibe'
7384 -> ' eine'
237233 -> ' Merge'
268 -> '-'
124070 -> 'Sort'
268 -> '-'
67699 -> 'Fun'
32813 -> 'ktion'
576 -> ' in'
6530 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 4893, 170242, 4842, 56840, 599, 3034, 34974, 714, 6530, 269, 4, 454]

=== Décodage token par token ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
4893 -> ' É'
170242 -> 'cris'
4842 -> ' une'
56840 -> ' fonction'
599 -> ' de'
3034 -> ' tri'
34974 -> ' fusion'
714 -> ' en'
6530 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

• Email: [email protected]
• Phone: +1 (516) 777-0945
• Location: New York, USA

📧 Copyright 2026 CMS Manhattan . All rights reserved