kaers-bert-241101

KAERS-BERT

Model Description

KAERS-BERT is a domain-specific Korean BERT model specialized for clinical text analysis, particularly for processing adverse drug event (ADE) narratives. It was developed by pretraining KoBERT (developed by SK Telecom) using 1.2 million ADE narratives reported through the Korea Adverse Event Reporting System (KAERS) between January 2015 and December 2019.

The model is specifically designed to handle clinical texts where code-switching between Korean and English is frequent, making it particularly effective for processing medical terms and abbreviations in a bilingual context.

Usage

You can load the model from HuggingFace Hub while using the local tokenizer:

from transformers import BertForPreTraining
from tokenization_kobert import KoBERTTokenizer

Load model from HuggingFace

model = BertForPreTraining.from_pretrained("kimsiun/kaers-bert-241101")

Load tokenizer from local file

tokenizer = KoBERTTokenizer.from_pretrained('skt/kobert-base-v1')

Key Features

• Specialized in clinical and pharmaceutical domain text
• Handles Korean-English code-switching common in medical texts
• Optimized for processing adverse drug event narratives
• Built upon KoBERT architecture with domain-specific pretraining

Training Data

The model was pretrained on:

• 1.2 million ADE narratives from KAERS
• Training data specifically focused on 'disease history in detail' and 'adverse event in detail' sections
• Masked language modeling with 15% token masking rate
• Maximum sequence length of 200
• Learning rate: 5×10^-5

Performance

The model demonstrated strong performance in various NLP tasks related to drug safety information extraction:

• Named Entity Recognition (NER): 83.81% F1-score
• Sentence Extraction: 76.62% F1-score
• Relation Extraction: 64.37% F1-score (weighted)
• Label Classification:
  • 'Occurred' Label: 81.33% F1-score
  • 'Concerned' Label: 77.62% F1-score

When applied to the KAERS database, the model achieved an average increase of 3.24% in data completeness for structured data fields.

Intended Use

This model is designed for:

• Extracting drug safety information from clinical narratives
• Processing Korean medical texts with English medical terminology
• Supporting pharmacovigilance activities
• Improving data quality in adverse event reporting systems

Limitations

• The model is specifically trained on adverse event narratives and may not generalize well to other clinical domains
• Performance may vary for texts significantly different from KAERS narratives
• The model works best with Korean clinical texts containing English medical terminology

Citation

@article{kim2023automatic,
  title={Automatic Extraction of Comprehensive Drug Safety Information from Adverse Drug Event Narratives in the Korea Adverse Event Reporting System Using Natural Language Processing Techniques},
  author={Kim, Siun and Kang, Taegwan and Chung, Tae Kyu and Choi, Yoona and Hong, YeSol and Jung, Kyomin and Lee, Howard},
  journal={Drug Safety},
  volume={46},
  pages={781--795},
  year={2023}