Downloads · 30 days
4
17% of all-time downloads
supertokenizer-mod_tokenizers_zero_padded
flexitok/supertokenizer-mod_tokenizers_zero_padded
supertokenizer-mod_tokenizers_zero_padded is a machine learning model from flexitok. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
A merged super-vocabulary built from 9 tokenizer(s).
All-time downloads
24
Public
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.8 MB · 100%
At a glance
- Access
- Public
- Created
- Mar 3, 2026
- Updated
- Mar 3, 2026
- SHA
- 7c08243b
Model card
From the Hugging Face model README
Super Vocabulary
A merged super-vocabulary built from 9 tokenizer(s).
Vocab size: 110931
Tokenizers
flexitok/mod-tokenizers-zero-padded-individualflexitok/mod-tokenizers-zero-padded-ltr_3digitflexitok/mod-tokenizers-zero-padded-ltr_2digitflexitok/mod-tokenizers-zero-padded-ltr_4digitflexitok/mod-tokenizers-zero-padded-ltr_5digitflexitok/mod-tokenizers-zero-padded-rtl_2digitflexitok/mod-tokenizers-zero-padded-rtl_3digitflexitok/mod-tokenizers-zero-padded-rtl_4digitflexitok/mod-tokenizers-zero-padded-rtl_5digit
Files
super_vocab.json— merged vocabulary mapping token string → super indexconfig.yaml— model config withvocab_sizeparticipating_tokenizers.json— list of tokenizer names included<tokenizer>_super_mapping.json— per-tokenizer index → super index mapping<tokenizer>_vocab.json— per-tokenizer vocabulary<tokenizer>_info.json/.yaml— tokenizer metadata
- Created
- Mar 3, 2026
- Updated
- Mar 3, 2026