Downloads · 30 days
0
FAM-tokenizer
Sergeial1972-user/FAM-tokenizer
FAM-tokenizer is a machine learning model from Sergeial1972-user. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Access
Public
Updated May 8, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.4 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- May 8, 2026
- Updated
- May 8, 2026
- SHA
- a5284706
Model card
From the Hugging Face model README
Model Card for Model ID
This is a tokenizer used in FAM model
Tokenizer Details
- Architecture: BPE
- Normalizer: BertNormalizer
- Pre-tokenizer: BertPreTokenizer
- Special tokens: [UNK], [PAD], [CLS], [SEP], [MASK]
- Dataset:
- Languages: English
Tokenizer training noteboook:
- Library
- transformers
- Created
- May 8, 2026
- Updated
- May 8, 2026