transformers-ud-japanese-bert-large-ginza

transformers-ud-japanese-bert-large-ginza (tohoku-nlp/bert-large-japanese-v2, fugashi, unidic-lite)

This is a bert-large model finetuned by spaCy v3 on UD_Japanese_BCCWJ r2.8.

The base pretrain model is tohoku-nlp/bert-large-japanese-v2, which requires fugashi and unidic-lite for tokenization.

The entire spaCy v3 model is distributed as a python package named ja_ginza_bert_large from PyPI along with GiNZA v5 which provides some custom pipeline components to recognize the Japanese bunsetu-phrase structures. Try running it as follows:

$ pip install ja-ginza-bert-large
$ ginza

Licenses

The models are distributed under the terms of the MIT License.

Acknowledgments

This model is permitted to be published under the MIT License under a joint research agreement between NINJAL (National Institute for Japanese Language and Linguistics) and Megagon Labs Tokyo.

Citation

@article{matsuda2020,
  title={GiNZA - Universal Dependencies による実用的日本語解析},
  author={松田 寛},
  journal={自然言語処理},
  volume={27},
  number={3},
  pages={695-701},
  year={2020},
  doi={10.5715/jnlp.27.695}
}

References

• UD_Japanese-BCCWJ

@inproceedings{asahara2018udjapanese,
    title = "{U}niversal {D}ependencies Version 2 for {J}apanese",
    author = "Asahara, Masayuki  and
      Kanayama, Hiroshi  and
      Tanaka, Takaaki  and
      Miyao, Yusuke  and
      Uematsu, Sumire  and
      Mori, Shinsuke  and
      Matsumoto, Yuji  and
      Omura, Mai  and
      Murawaki, Yugo",
    booktitle = "Proceedings of the Eleventh International Conference on Language Resources and Evaluation ({LREC} 2018)",
    month = may,
    year = "2018",
    address = "Miyazaki, Japan",
    publisher = "European Language Resources Association (ELRA)",
    url = "https://aclanthology.org/L18-1287/"
}