RoBERTa-base-janpanese

This is RoBERTa model pretrained on texts in the Japanese language.

3.45GB wikipedia text

trained 1.65M step

use the sentencepiece tokenizer.

If you want to fine-tune model. Please use

from transformers import BertTokenizer, RobertaModel
BertTokenizer.from_pretrained('')
RoBERTModel.from_pretrained('')

The accuracy in JGLUE-marc_ja-v1.0 binary sentiment classification 95.4%

Contribute by Yokohama Nationaly University Mori Lab

@article{liu2019roberta,
title={Roberta: A robustly optimized bert pretraining approach},
author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
journal={arXiv preprint arXiv:1907.11692},
year={2019}
}