Downloads · 30 days
272
39% of all-time downloads
japanese-dummy-tokenizer
ybelkada/japanese-dummy-tokenizer
japanese-dummy-tokenizer is a machine learning model from ybelkada. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
Repository containing a dummy Japanese Tokenizer trained on dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner.
All-time downloads
699
Public
Repo size
—
Likes
1
Public
Repo makeup
Click a slice to open those files.
.json4.1 MB · 100%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Apr 6, 2022
- Updated
- Jul 11, 2022
- SHA
- 5d769738
Try a prompt
Datasets
Model card
From the Hugging Face model README
Japanese Dummy Tokenizer
Repository containing a dummy Japanese Tokenizer trained on snow_simplified_japanese_corpus dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner.
Intended uses & limitations
You can use this tokenizer to tokenize Japanese sentences.
How to use it
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ybelkada/japanese-dummy-tokenizer")
How to train the tokenizer
Check the file tokenizer.py, you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from csebuetnlp/mT5_multilingual_XLSum.
- Library
- transformers
- License
- mit
- Languages
- en, ja
- Created
- Apr 6, 2022
- Updated
- Jul 11, 2022