dialog-koelectra-small-discriminator

Dialog-KoELECTRA

Github : https://github.com/skplanet/Dialog-KoELECTRA

Introduction

Dialog-KoELECTRA is a language model specialized for dialogue. It was trained with 22GB colloquial and written style Korean text data. Dialog-ELECTRA model is made based on the ELECTRA model. ELECTRA is a method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN. At small scale, ELECTRA achieves strong results even when trained on a single GPU.

<br>

Released Models

We are initially releasing small version pre-trained model. The model was trained on Korean text. We hope to release other models, such as base/large models, in the future.

<br>

Model Performance

Dialog-KoELECTRA shows strong performance in conversational downstream tasks.

<br>

Train Data

<table class="tg"> <thead> <tr> <th class="tg-c3ow"></th> <th class="tg-c3ow">corpus name</th> <th class="tg-c3ow">size</th> </tr> </thead> <tbody> <tr> <td class="tg-c3ow" rowspan="4">dialog</td> <td class="tg-0pky"><a href="https://aihub.or.kr/aidata/85" target="_blank" rel="noopener noreferrer">Aihub Korean dialog corpus</a></td> <td class="tg-c3ow" rowspan="4">7GB</td> </tr> <tr> <td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Spoken corpus</a></td> </tr> <tr> <td class="tg-0pky"><a href="https://github.com/songys/Chatbot_data" target="_blank" rel="noopener noreferrer">Korean chatbot data</a></td> </tr> <tr> <td class="tg-0pky"><a href="https://github.com/Beomi/KcBERT" target="_blank" rel="noopener noreferrer">KcBERT</a></td> </tr> <tr> <td class="tg-c3ow" rowspan="2">written</td> <td class="tg-0pky"><a href="https://corpus.korean.go.kr/" target="_blank" rel="noopener noreferrer">NIKL Newspaper corpus</a></td> <td class="tg-c3ow" rowspan="2">15GB</td> </tr> <tr> <td class="tg-0pky"><a href="https://github.com/lovit/namuwikitext" target="_blank" rel="noopener noreferrer">namuwikitext</a></td> </tr> </tbody> </table> <br>

Vocabulary

We applied morpheme analysis using huggingface_konlpy when creating a vocabulary dictionary. As a result of the experiment, it showed better performance than a vocabulary dictionary created without applying morpheme analysis.

<table> <thead> <tr> <th>vocabulary size</th> <th>unused token size</th> <th>limit alphabet</th> <th>min frequency</th> </tr> </thead> <tbody> <tr> <td>40,000</td> <td>500</td> <td>6,000</td> <td>3</td> </tr> </tbody> </table> <br>