Downloads · 30 days
0
0925_bigbench_code_tokenizer
thepowerfuldeez/0925_bigbench_code_tokenizer
0925_bigbench_code_tokenizer is a machine learning model from thepowerfuldeez. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
SuperBPE tokenizer trained on mix of code + edu data
Access
Public
Updated Sep 25, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json4.6 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Sep 25, 2025
- Updated
- Sep 25, 2025
- SHA
- b466b9a4
Model card
From the Hugging Face model README
SuperBPE tokenizer
SuperBPE tokenizer trained on mix of code + edu data
"Big cat sat on a mat" -> ['Big', 'Ġcat', 'Ġsat', 'ĠonĠa', 'Ġmat']
"this function does summation on a last axis and then returns a scalar." -> ['this', 'Ġfunction', 'Ġdoes', 'Ġsummation', 'ĠonĠa', 'Ġlast', 'Ġaxis', 'ĠandĠthen', 'ĠreturnsĠa', 'Ġscalar', '.']
- Library
- transformers
- Created
- Sep 25, 2025
- Updated
- Sep 25, 2025