AliBERT-7GB

quinten-datalab/AliBERT-7GB: AliBERT: is a pre-trained language model for French biomedical text.

Introduction

AliBERT: is a pre-trained language model for French biomedical text. It is trained with masked language model like RoBERTa.

Here are the main contributions of our work:

<ul> <li> A French biomedical language model, a language-specific and domain-specific PLM, which can be used to represent French biomedical text for different downstream tasks. </li> <li> A normalization of a Unigram sub-word tokenization of French biomedical textual input which improves our vocabulary and overall performance of the models trained. </li> <li> It is a foundation model that achieved state-of-the-art results on French biomedical text. </li> </ul>

The Paper can be found here: https://aclanthology.org/2023.bionlp-1.19/

Data

The pre-training corpus was gathered from different sub-corpora. It is composed of 7GB French biomedical textual documents. The corpora were collected from different sources. Scientific articles are collected from ScienceDirect using an API provided on subscription and where French articles in biomedical domain were selected. The summaries of thesis manuscripts are collected from "Système universitaire de documentation (SuDoc)" which is a catalog of universities documentation system. Short texts and some complete sentences were collected from the public drug database which lists the characteristics of tens of thousands of drugs. Furthermore, a similar drug database known as "Résumé des Caractéristiques du Produit (RCP)" is also used to represent a description of medications that are intended to be utilized by biomedicine professionals.

How to use alibert-quinten/Oncology-NER with HuggingFace

Load quinten-datalab/AliBERT-7GB fill-mask model and the tokenizer used to train AliBERT:

from transformers import AutoTokenizer, AutoModelForTokenClassification,pipeline

tokenizer = AutoTokenizer.from_pretrained("quinten-datalab/AliBERT-7GB")

model = AutoModelForTokenMaskedLM.from_pretrained("quinten-datalab/AliBERT-7GB")

fill_mask=pipeline("fill-mask",model=model,tokenizer=tokenizer)
nlp_AliBERT=fill_mask("La prise de greffe a été systématiquement réalisée au niveau de la face interne de la [MASK] afin de limiter la plaie cicatricielle.")

[{'score': 0.7724128365516663,
  'token': 6749,
  'token_str': 'cuisse',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la cuisse afin de limiter la plaie cicatricielle.'},
 {'score': 0.09472355246543884,
  'token': 4915,
  'token_str': 'jambe',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la jambe afin de limiter la plaie cicatricielle.'},
 {'score': 0.03340734913945198,
  'token': 2050,
  'token_str': 'main',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la main afin de limiter la plaie cicatricielle.'},
 {'score': 0.030924487859010696,
  'token': 844,
  'token_str': 'face',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la face afin de limiter la plaie cicatricielle.'},
 {'score': 0.012518334202468395,
  'token': 3448,
  'token_str': 'joue',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la joue afin de limiter la plaie cicatricielle.'}]

Metrics and results

The model has been evaluted in the following downstream tasks

Biomedical Named Entity Recognition (NER)

The model is evaluated on two (CAS and QUAERO) publically available Frech biomedical text.

CAS dataset

<style type="text/css"> .tg {border-collapse:collapse;border-spacing:0;} .tg .tg-baqh{text-align:center;vertical-align:top} .tg .tg-0lax{text-align:center;vertical-align:top} </style> <table class="tg"> <thead> <tr> <th>Models</th> <th class="tg-0lax" colspan="3">CamemBERT</th> <th class="tg-0lax" colspan="3">AliBERT</th> <th class="tg-0lax" colspan="3">DrBERT</th> </tr> </thead> <tbody> <tr> <td>Entities</td><td>P<br></td><td>R</td><td>F1</td><td>P<br></td><td>R</td><td>F1</td><td>P<br></td><td>R</td><td>F1</td> </tr> <tr> <td>Substance</td><td>0.96</td><td>0.87</td><td>0.91</td><td>0.96</td><td>0.91</td><td>0.93</td><td>0.83</td><td>0.83</td><td>0.82</td> </tr> <tr> <td>Symptom</td> <td>0.89</td> <td>0.91</td> <td>0.90</td> <td>0.96</td> <td>0.98</td> <td>0.97</td> <td>0.93</td> <td>0.90</td> <td>0.91</td> </tr> <tr> <td>Anatomy</td> <td>0.94</td> <td>0.91</td> <td>0.88</td> <td>0.97</td> <td>0.97</td> <td>0.98</td> <td>0.92</td> <td>0.93</td> <td>0.93</td> </tr> <tr> <td>Value</td> <td>0.88</td> <td>0.46</td> <td>0.60</td> <td>0.98</td> <td>0.99</td> <td>0.98</td> <td>0.91</td> <td>0.91</td> <td>0.91</td> </tr> <tr> <td> Pathology</td> <td>0.79</td> <td>0.70</td> <td>0.74</td> <td>0.81</td> <td>0.39</td> <td>0.52</td> <td>0.85 <td>0.57</td> <td>0.68</td> </tr> <tr> <td>Macro Avg</td> <td>0.89 </td> <td>0.79</td> <td>0.81</td> <td> 0.94</td> <td>0.85</td> <td>0.88</td> <td> 0.92</td> <td> 0.87</td> <td>0.89</td> </tr> </tbody> </table> Table 1: NER performances on CAS dataset

QUAERO dataset

<table class="tg"> <thead> <tr> <th>Models</th> <th class="tg-0lax" colspan="3">CamemBERT</th> <th class="tg-0lax" colspan="3">AliBERT</th> <th class="tg-0lax" colspan="3">DrBERT</th> </tr> </thead> <tbody> <tr> <td>Entity </td> <td> P </td> <td> R </td> <td> F1 </td> <td> P </td> <td> R </td> <td> F1 </td> <td> P </td> <td> R </td> <td> F1 </td> </tr> <tr> <td>Anatomy </td> <td> 0.649 </td> <td> 0.641 </td> <td> 0.645 </td> <td> 0.795 </td> <td> 0.811 </td> <td> 0.803 </td> <td> 0.736 </td> <td> 0.844 </td> <td> 0.824 </td> </tr> <tr> <td>Chemical </td> <td> 0.844 </td> <td> 0.847 </td> <td> 0.846 </td> <td> 0.878 </td> <td> 0.893 </td> <td> 0.885 </td> <td> 0.505 </td> <td> 0.823 </td> <td> 0.777 </td> </tr> <tr> <td>Device </td> <td> 0.000 </td> <td> 0.000 </td> <td> 0.000 </td> <td> 0.506 </td> <td> 0.356 </td> <td> 0.418 </td> <td> 0.939 </td> <td> 0.237 </td> <td> 0.419 </td> </tr> <tr> <td>Disorder </td> <td> 0.772 </td> <td> 0.818 </td> <td> 0.794 </td> <td> 0.857 </td> <td> 0.843 </td> <td> 0.850 </td> <td> 0.883 </td> <td> 0.809 </td> <td> 0.845 </td> </tr> <tr> <td>Procedure </td> <td> 0.880 </td> <td> 0.894 </td> <td> 0.887 </td> <td> 0.969 </td> <td> 0.967 </td> <td> 0.968 </td> <td> 0.944 </td> <td> 0.976 </td> <td> 0.960 </td> </tr> <tr> <td>Macro Avg </td> <td> 0.655 </td> <td> 0.656 </td> <td> 0.655 </td> <td> 0.807 </td> <td> 0.783 </td> <td> 0.793 </td> <td> 0.818 </td> <td> 0.755 </td> <td> 0.782 </td> </tr> </tbody> </table> Table 2: NER performances on QUAERO dataset

##AliBERT: A Pre-trained Language Model for French Biomedical Text