Downloads · 30 days
0
devngho/phi3-jamo-ko-large-tokenizer
phi3-jamo-ko-large-tokenizer is a machine learning model from devngho. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Downloads · 30 days
0
Access
Public
Updated Aug 25, 2024
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json7.6 MB · 100%
From the Hugging Face model README
expanded 49556 tokens
Following The Optimal Vocabulary Size Predictor, I recommend using this tokenizer with 13B model.