Downloads · 30 days
0
turkish-bigbird-tokenizer
FiratIsmailoglu/turkish-bigbird-tokenizer
turkish-bigbird-tokenizer is a machine learning model from FiratIsmailoglu. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
It is a sentencepice tokenizer trained using the Oscar Turkish Corpus from scratch. The tokenizer can be used for bigbird models and can be callled either typing AutoTokenizer.frompretrained("FiratIsmailoglu/turkish-b…
Access
Public
Updated Feb 2, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.4 MB · 100%
At a glance
- Access
- Public
- Created
- Feb 2, 2025
- Updated
- Feb 2, 2025
- SHA
- 9d55e9d0
Model card
From the Hugging Face model README
It is a sentencepice tokenizer trained using the Oscar Turkish Corpus from scratch. The tokenizer can be used for bigbird models and can be callled either typing AutoTokenizer.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer") or BigBirdTokenizerFast.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer").
- Created
- Feb 2, 2025
- Updated
- Feb 2, 2025