distilbert-base-uncased-FT-ner-BC5CDR-disease

Model Card for Model ID

Fine-tuned distilbert model. Trained on train set of BC5CDR-disease dataset taken from BLURB.

Model Details

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/kbulutozler/medical-llm-benchmark

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Train set of BC5CDR-disease dataset.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Classical fine-tuning.

Training Hyperparameters

• Training regime: [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

learning_rate=5e-5 per_device_train_batch_size=16 per_device_eval_batch_size=16 num_train_epochs=3 weight_decay=0.01

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data

<!-- This should link to a Dataset Card if possible. -->

Test set of BC5CDR-disease dataset.

Results

Precision: 0.76 Recall: 0.81 Micro-F1: 0.79

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

• Hardware Type: 1xRTX A4000
• Hours used: 00:07:00