Multi-super-128K

tinycompany/Multi-super-128K

Multi-super-128K is a machine learning model from tinycompany. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.

Downloads · 30 days

Access

Public

Updated Apr 2, 2025

Repo size

21.6 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.json10.8 MB · 100%

At a glance

Library: transformers
Access: Public
Created: Apr 2, 2025
Updated: Apr 2, 2025
SHA: 3276d1c3

Spaces

eduagarcia/multilingual-tokenizer-leaderboard

Model card

From the Hugging Face model README

	Tokenizer	English	Hindi	Tamil	Bengali	Kannad	Malayalam	Telugu	Gujarati	Punjabi	Code_Python	Code_Java	c++	Math
0	fhai50032/BPE-Multilingual-72k (72k)	343695 reversible	13684 reversible	46603 reversible	30681 reversible	2695 reversible	34103 reversible	36488 reversible	89326 reversible	76027 irreversible, 2 words differ	5367 reversible	2269 reversible	13072 reversible	5026 reversible
1	tinycompany/Multi-super-128K (128k)	306753 reversible	11874 reversible	39278 reversible	28020 reversible	2492 reversible	31887 reversible	31561 reversible	75840 reversible	60832 irreversible, 2 words differ	4790 reversible	1768 reversible	10828 reversible	3464 reversible
2	unsloth/gemma-3-27b-it-unsloth-bnb-4bit (262k)	329308 reversible	10833 reversible	50419 reversible	21846 reversible	3279 reversible	33152 reversible	44481 reversible	100232 reversible	83628 reversible	5998 reversible	2590 reversible	15824 reversible	5867 reversible
3	HuggingFaceTB/SmolLM2-1.7B-Instruct (49k)	337029 reversible	44629 reversible	76096 reversible	173200 reversible	19741 reversible	195612 reversible	69749 reversible	162463 reversible	110620 reversible	5873 reversible	2548 reversible	15702 reversible	6200 reversible
4	fhai50032/QTK-81K (81k)	318160 reversible	10767 reversible	65787 reversible	152278 reversible	14877 reversible	171650 reversible	53429 reversible	129691 reversible	94554 irreversible, 2 words differ	5192 reversible	2241 reversible	12730 reversible	5120 reversible
5	deepseek-ai/DeepSeek-R1 (128k)	338873 reversible	22854 reversible	49138 reversible	39616 reversible	5680 reversible	74228 reversible	40360 reversible	100608 reversible	78854 reversible	5270 reversible	2223 reversible	12785 reversible	5375 reversible
6	unsloth/phi-4 (100k)	308645 reversible	40456 reversible	59899 reversible	116122 reversible	12642 reversible	150518 reversible	48706 reversible	117951 reversible	87107 reversible	4844 reversible	2110 reversible	12088 reversible	5573 reversible
7	deepseek-ai/DeepSeek-R1-Distill-Llama-8B (128k)	308511 reversible	21109 reversible	59773 reversible	115137 reversible	12642 reversible	150511 reversible	48677 reversible	117676 reversible	86457 reversible	4843 reversible	2110 reversible	12088 reversible	5573 reversible
8	unsloth/gemma-2-9b-it(256k)	323334 reversible	15915 reversible	54109 reversible	53401 reversible	5302 reversible	57427 reversible	47641 reversible	107516 reversible	86795 reversible	5986 reversible	2568 reversible	15806 reversible	5870 reversible
9	krutrim-ai-labs/Krutrim-2-instruct (131k)	343067 reversible	15477 reversible	47456 reversible	38425 reversible	3637 reversible	46206 reversible	38865 reversible	92455 reversible	76397 reversible	5246 reversible	2223 reversible	12700 reversible	5361 reversible
10	sarvamai/sarvam-2b-v0.5 (68k)	385383 reversible	11256 reversible	61652 reversible	27347 reversible	2532 reversible	31919 reversible	51500 reversible	119103 reversible	102859 reversible	7387 reversible	3067 reversible	17586 reversible	6863 reversible

Library: transformers
Created: Apr 2, 2025
Updated: Apr 2, 2025