Downloads · 30 days
0
0% of all-time downloads
sentencepiece_ja
if001/sentencepiece_ja
sentencepiece_ja is a machine learning model from if001. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-4.0.
sentencepiece unigramを日本語で学習 https://github.com/huggingface/tokenizers
All-time downloads
8
Public
Repo size
—
Likes
2
Public
Repo makeup
Click a slice to open those files.
.json2.2 MB · 89%
At a glance
- License
- cc-by-4.0
- Access
- Public
- Created
- Sep 27, 2023
- Updated
- Jan 23, 2024
- SHA
- e47ab470
Model card
From the Hugging Face model README
sentencepiece unigramを日本語で学習
https://github.com/huggingface/tokenizers
sample
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("if001/sentencepiece_ja", trust_remote_code=True)
print(tokenizer("hello world"))
>> {'input_ids': [158, 8418, 1427, 15930, 866, 13782, 44, 15034, 1719, 16655, 8, 115, 5, 280, 17635, 94, 818, 2748, 1168, 1114], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
print(tokenizer.tokenize('それは九月初旬のある蒸し暑い晩のことであった。私は、Ｄ坂の大通りの中程にある'))
>> ['それは', '九月', '初', '旬', 'のある', '蒸', 'し', '暑い', '晩', 'のことであった', '。', '私は', '、', 'D', '坂の', '大', '通り', 'の中', '程', 'にある']
データセット
https://huggingface.co/datasets/izumi-lab/wikinews-ja-20230728
https://huggingface.co/datasets/izumi-lab/wikinews-en-20230728
https://huggingface.co/datasets/if001/aozorabunko-clean-sin
settings
all_special_ids: [1, 2, 3, 0, 4]
all_special_tokens: ['<BOS>', '<EOS>', '<UNK>', '<PAD>', '<MASK>']
- License
- cc-by-4.0
- Languages
- en, ja
- Created
- Sep 27, 2023
- Updated
- Jan 23, 2024