Downloads · 30 days
0
BPE-40k_OSCAR-en-30M
Bauwens/BPE-40k_OSCAR-en-30M
BPE-40k_OSCAR-en-30M is a machine learning model from Bauwens. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
BPE tokeniser with vocabulary size 40000, trained on the first 30 million examples in English OSCAR.
Access
Public
Updated Dec 3, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.6 MB · 87%
At a glance
- Access
- Public
- Created
- Dec 3, 2025
- Updated
- Dec 3, 2025
- SHA
- bc2057c8
Model card
From the Hugging Face model README
BPE-40k OSCAR-en-30M
BPE tokeniser with vocabulary size 40000, trained on the first 30 million examples in English OSCAR.
Tokeniser details
BPE trainer implementation (note that these are long deprecated and you should use TkTkT):
- Back-end: HuggingFace
tokenizers'sBpeTrainer. - Front-end: BPE-knockout's
BPETrainer
Preprocessor: see the appendix of the BPE-knockout paper.
Training details
Time: 3h10m
- Preprocessing and counting the 30M corpus: 28h30m
- BPE merges: under 2 hours
Data sizes:
- Examples used: 30 000 000
- Created
- Dec 3, 2025
- Updated
- Dec 3, 2025