binary_icf_classifier

A-PROOF Binary Sentence Classification

Description

A fine-tuned binary text classification model that determines whether a sentence is relevant for WHO-ICF category classification.

Since 95% of the sentences in clinical notes is not relevant for ICF classification, it makes sense to filter the relevant sentences before applying other classification processes. Using the binary classification, the processing of large volumes of data can be optimised as only 5% of the sentences needs to be classified for the level of functioning. For further classification of relevant sentences, you can use the multilabel classifier: https://huggingface.co/CLTL/icf-domains and the any of the relevant regression classifiers for obtaining a level score.

Relevant sentences are likely to be express patient's functioning for the following 9 ICF categories:

Intended use and limitations

• The model was fine-tuned (trained, validated and tested) on medical records from the Amsterdam UMC (the two academic medical centers of Amsterdam). It might perform differently on text from a different hospital or text from non-hospital sources (e.g. GP records).
• The model only distinguishes sentences on the basis of the 9 ICF categories.

How to use

To generate predictions with the model, use the Transformers library:

# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline('text-classification', model='CLTL/binary_icf_classifier')
result = pipe('De patient is erg moe')
print(result)
[{'label': 'pos', 'score': 0.9977788329124451}]

# load model directly
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer. from_pretrained ('CLTL/binary_icf_classifier')
model = AutoModelForSequenceClassification.from_pretrained('CLTL/binary_icf_classifier')

Training data

• The training data consists of clinical notes from medical records (in Dutch) of the Amsterdam UMC. Due to privacy constraints, the data cannot be released.
• The annotation guidelines used for the project can be found here.

Evaluation results

The evaluation is done on a sentence-level (the classification unit): .97 precision, .96 recall, .97 f1.

Contact

Piek Vossen, piek.vossen@vu.nl

References

https://github.com/cltl-students/Cecilia_Kuan_data_augmentation Cecilia Kuan, 2023, Generative Approach of Data Augmentation for Pre-Trained Clinical NLP System, MA Thesis, Vrije Universiteit Amsterdam