bart-ko-base

Pretrained BART in Korean

This is pretrained BART model with multiple Korean Datasets.

I used multiple datasets for generalizing the model for both colloquial and written texts.

The training is supported by TPU Research Cloud program.

The script which is used to pre-train model is here.

When you use the reference API, you must wrap the sentence with [BOS] and [EOS] like below example.

[BOS] 안녕하세요? 반가워요~~ [EOS]

You can also test mask filling performance using [MASK] token like this.

[BOS] [MASK] 먹었어? [EOS]

Benchmark

<style> table { border-collapse: collapse; border-style: hidden; width: 100%; } td, th { border: 1px solid #4d5562; padding: 8px; } </style> <table> <tr> <th>Dataset</th> <td>KLUE NLI dev</th> <td>NSMC test</td> <td>QuestionPair test</td> <td colspan="2">KLUE TC dev</td> <td colspan="3">KLUE STS dev</td> <td colspan="3">KorSTS dev</td> <td colspan="2">HateSpeech dev</td> </tr> <tr> <th>Metric</th> <!-- KLUE NLI --> <td>Acc</th> <!-- NSMC --> <td>Acc</td> <!-- QuestionPair --> <td>Acc</td> <!-- KLUE TC --> <td>Acc</td> <td>F1</td> <!-- KLUE STS --> <td>F1</td> <td>Pearson</td> <td>Spearman</td> <!-- KorSTS --> <td>F1</td> <td>Pearson</td> <td>Spearman</td> <!-- HateSpeech --> <td>Bias Acc</td> <td>Hate Acc</td> </tr> <tr> <th>Score</th> <!-- KLUE NLI --> <td>0.7390</th> <!-- NSMC --> <td>0.8877</td> <!-- QuestionPair --> <td>0.9208</td> <!-- KLUE TC --> <td>0.8667</td> <td>0.8637</td> <!-- KLUE STS --> <td>0.7654</td> <td>0.8090</td> <td>0.8040</td> <!-- KorSTS --> <td>0.8067</td> <td>0.7909</td> <td>0.7784</td> <!-- HateSpeech --> <td>0.8280</td> <td>0.5669</td> </tr> </table>

• The performance was measured using the notebooks here with colab.

Used Datasets

모두의 말뭉치

• 일상 대화 말뭉치 2020
• 구어 말뭉치
• 문어 말뭉치
• 신문 말뭉치

AIhub

• 개방데이터 전문분야말뭉치
• 개방데이터 한국어대화요약
• 개방데이터 감성 대화 말뭉치
• 개방데이터 한국어 음성
• 개방데이터 한국어 SNS

세종 말뭉치