Downloads · 30 days
1.4K
0% of all-time downloads
tohoku-nlp/bert-base-japanese-char-whole-word-masking
bert-base-japanese-char-whole-word-masking is a fill-mask model from tohoku-nlp. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as cc-by-sa-4.0.
This is a BERT model pretrained on texts in the Japanese language.
Downloads · 30 days
1.4K
0% of all-time downloads
All-time downloads
360K
Public
Repo size
1.8 GB
Likes
4
Public
Click a slice to open those files.
.h5373 MB · 34%
From the Hugging Face model README
This is a BERT model pretrained on texts in the Japanese language.
This version of the model processes input texts with word-level tokenization based on the IPA dictionary, followed by character-level tokenization. Additionally, the model is trained with the whole word masking enabled for the masked language modeling (MLM) objective.
The codes for the pretraining are available at cl-tohoku/bert-japanese.
The model architecture is the same as the original BERT base model; 12 layers, 768 dimensions of hidden states, and 12 attention heads.
The model is trained on Japanese Wikipedia as of September 1, 2019. To generate the training corpus, WikiExtractor is used to extract plain texts from a dump file of Wikipedia articles. The text files used for the training are 2.6GB in size, consisting of approximately 17M sentences.
The texts are first tokenized by MeCab morphological parser with the IPA dictionary and then split into characters. The vocabulary size is 4000.
The model is trained with the same configuration as the original BERT; 512 tokens per instance, 256 instances per batch, and 1M training steps.
For the training of the MLM (masked language modeling) objective, we introduced the Whole Word Masking in which all of the subword tokens corresponding to a single word (tokenized by MeCab) are masked at once.
The pretrained models are distributed under the terms of the Creative Commons Attribution-ShareAlike 3.0.
For training models, we used Cloud TPUs provided by TensorFlow Research Cloud program.