Downloads · 30 days
0
finewebedu_32000
gvlassis/finewebedu_32000
finewebedu_32000 is a machine learning model from gvlassis. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
🇬🇧 An English tokenizer, trained on the FineWeb-Edu dataset.
Access
Public
Updated Sep 5, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.3 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024
- SHA
- 3d5971c5
Model card
From the Hugging Face model README
finewebedu_32000
About
🇬🇧 An English tokenizer, trained on the FineWeb-Edu dataset.
Description
This is a character-level (mainly) English (en) tokenizer, trained on the CC-MAIN-2024-10 subset of FineWeb-Edu. It has a vocabulary size of 32,000 (multiple of 128), which makes it fast for integration in models.
Usage
import tokenizers
dataset = tokenizers.Tokenizer.from_pretrained("gvlassis/finewebedu_32000")
- Library
- transformers
- Created
- Sep 5, 2024
- Updated
- Sep 5, 2024