Downloads · 30 days
0
slimpajama_tok-48128-BPE-forT5
BEE-spoke-data/slimpajama_tok-48128-BPE-forT5
slimpajama_tok-48128-BPE-forT5 is a machine learning model from BEE-spoke-data. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
custom tokenizer from scaling study adapted for T5 training
Access
Public
Updated Dec 29, 2025
Repo size
1 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json3.2 MB · 75%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Aug 7, 2024
- Updated
- Dec 29, 2025
- SHA
- d20b22c5
Model card
From the Hugging Face model README
48k vocab LlamaTokenizer for T5
custom tokenizer from scaling study adapted for T5 training
- Compression ratio: 3.54
- Vocabulary size: 48228
Tokens:
['▁In', '▁', '2', '0', '2', '3', ',', '▁Dr', '.', '▁Jane', '▁Smith', '-', 'John', 'son', '▁published', '▁groundbreaking', '▁research', '▁on', '▁quantum', '▁ent', 'ang', 'lement', ',', '▁demonstrating', '▁a', '▁', '9', '9', '.', '9', '%', '▁success', '▁rate', '▁in', '▁tele', 'port', 'ing', '▁qu', 'bits', '▁over', '▁', '1', '0', '0', 'km', '▁using', '▁her', '▁patented', "▁'", 'Q', '-', 'Link', "'", '▁technology', '.', '</s>']
- Library
- transformers
- License
- mit
- Languages
- en
- Created
- Aug 7, 2024
- Updated
- Dec 29, 2025