Downloads · 30 days
2
7% of all-time downloads
supertokenizer-maddition
flexitok/supertokenizer-maddition
supertokenizer-maddition is a machine learning model from flexitok. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
A merged super-vocabulary built from 22 tokenizer(s).
All-time downloads
30
Public
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json638 KB · 99%
At a glance
- Access
- Public
- Created
- Apr 18, 2026
- Updated
- Apr 18, 2026
- SHA
- 0e6f553c
Model card
From the Hugging Face model README
Super Vocabulary
A merged super-vocabulary built from 22 tokenizer(s).
Vocab size: 13290
Tokenizers
flexitok/maddition_jpn_Jpan_2000flexitok/maddition_cmn_Hani_2000flexitok/maddition_ita_Latn_2000flexitok/maddition_por_Latn_2000flexitok/maddition_swe_Latn_2000flexitok/maddition_deu_Latn_2000flexitok/maddition_ind_Latn_2000flexitok/maddition_fas_Arab_2000flexitok/maddition_vie_Latn_2000flexitok/maddition_nld_Latn_2000flexitok/maddition_fra_Latn_2000flexitok/maddition_spa_Latn_2000flexitok/maddition_dan_Latn_2000flexitok/maddition_eng_Latn_2000flexitok/maddition_arb_Arab_2000flexitok/maddition_ces_Latn_2000flexitok/maddition_hun_Latn_2000flexitok/maddition_tur_Latn_2000flexitok/maddition_pol_Latn_2000flexitok/maddition_ell_Grek_2000flexitok/maddition_rus_Cyrl_2000flexitok/maddition_digit_2000
Files
super_vocab.json— merged vocabulary mapping token string → super indexconfig.yaml— model config withvocab_sizeparticipating_tokenizers.json— list of tokenizer names included<tokenizer>_super_mapping.json— per-tokenizer index → super index mapping<tokenizer>_vocab.json— per-tokenizer vocabulary<tokenizer>_info.json/.yaml— tokenizer metadata
- Created
- Apr 18, 2026
- Updated
- Apr 18, 2026