Downloads · 30 days
291
8% of all-time downloads
izumi-lab/electra-small-paper-japanese-fin-discriminator
electra-small-paper-japanese-fin-discriminator is a machine learning model from izumi-lab. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as cc-by-sa-4.0.
This is a ELECTRA model pretrained on texts in the Japanese language.
Downloads · 30 days
291
8% of all-time downloads
All-time downloads
3.6K
Public
Repo size
55.4 MB
Likes
0
Public
Click a slice to open those files.
.bin55.4 MB · 100%
From the Hugging Face model README
This is a ELECTRA model pretrained on texts in the Japanese language.
The codes for the pretraining are available at retarfi/language-pretraining.
The model architecture is the same as ELECTRA small in the original ELECTRA paper; 12 layers, 256 dimensions of hidden states, and 4 attention heads.
The models are trained on the Japanese version of Wikipedia.
The training corpus is generated from the Japanese version of Wikipedia, using Wikipedia dump file as of June 1, 2021.
The Wikipedia corpus file is 2.9GB, consisting of approximately 20M sentences.
The financial corpus consists of 2 corpora:
Summaries of financial results from October 9, 2012, to December 31, 2020
Securities reports from February 8, 2018, to December 31, 2020
The financial corpus file is 5.2GB, consisting of approximately 27M sentences.
The texts are first tokenized by MeCab with IPA dictionary and then split into subwords by the WordPiece algorithm.
The vocabulary size is 32768.
The models are trained with the same configuration as ELECTRA small in the original ELECTRA paper; 128 tokens per instance, 128 instances per batch, and 1M training steps.
@article{Suzuki-etal-2023-ipm,
title = {Constructing and analyzing domain-specific language model for financial text mining}
author = {Masahiro Suzuki and Hiroki Sakaji and Masanori Hirano and Kiyoshi Izumi},
journal = {Information Processing & Management},
volume = {60},
number = {2},
pages = {103194},
year = {2023},
doi = {10.1016/j.ipm.2022.103194}
}
The pretrained models are distributed under the terms of the Creative Commons Attribution-ShareAlike 4.0.
This work was supported by JSPS KAKENHI Grant Number JP21K12010.