Downloads · 30 days
0
gururaser/bilimkurgu-bpe-tokenizer
bilimkurgu-bpe-tokenizer is a machine learning model from gururaser. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-nc-4.0.
gururaser/ithaki-bilimkurgu-klasikleri veri setindeki kitap adı, yazar ve özet metinlerinden eğitilmiş byte-level BPE tokenizer. Vocab boyutu: 1000.
Downloads · 30 days
0
Access
Public
Updated Jul 19, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json63.3 KB · 97%
From the Hugging Face model README
gururaser/ithaki-bilimkurgu-klasikleri veri setindeki
kitap adı, yazar ve özet metinlerinden eğitilmiş byte-level BPE tokenizer.
Vocab boyutu: 1000.
Eğitim amaçlı oluşturulmuştur.
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
tokenizer.encode("Isaac Asimov").tokens