sentence-level-ignorance-classifier

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This model is a binary classifier that determines whether a sentence is a statement of ignorance or not.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• This model consists of a linear classification head on top of BioMedBERT, which is a BERT-based encoder (110 million parameters, 12 layers, 768 hidden).

• All layers besides 10, 11, the pooler, and classificaton head were frozen to reduce overfitting on the small training set.

• Developed by: Nathan Gelfand, Darya Shlyk

• Funded by: Larry Hunter

• Model type: Supervised fine-tune of a transformer-encoder for text classification

• Language(s) (NLP): English

• License: MIT

• Finetuned from model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: [More Information Needed]

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

A statement of ignorance, or knowledge gap statement, is a piece of text indicating a lack of knoweldge on some subject. In the field of biomedical research, knowledge gap statements often indicate unanswered questions, which can be useful for researchers looking for new topics of study. We built this sentence-level binary classifier so that researchers could parse through biomedical papers to identify research directives.

<!-- ### Downstream Use [optional] <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed] -->

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

This model was trained on a dataset of 937 annotated statements from articles about monogenic epilepsies. Due to the small dataset size, it is possible that the classifier will perform better on statements related to monogenic epilepsies.

The annotated statements were based on the guidelines below. Therefore, the model's classifications will also be restricted to these guidelines. annotation_guidelines.html

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

!pip install transformers 
from transformers import pipeline

classifier = pipeline("text-classification",model="hunter-lab/sentence-level-ignorance-classifier")

sentence = "Consequently, there is a pressing need to enhance basic mechanistic and clinical studies, with a focus on individual differences, to advance the exploration of BCAAs in treating neurological disorders and to provide more robust evidence supporting their clinical applications."

result = classifier(sentence)

print(result)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Our training data consists of 937 statements from articles extracted from PubMed that are related to monogenic epilepsies (fill out later). Each statement was annotated as "ignorance" or "not ignorance" based on the guidelines in annotation_guidelines.html.
There are 745 training statements, 104 validation statements, and 88 testing statements.

Training Procedure

We split the data roughly 80-10-10 into train, test, and validation sets, grouping statements by article PMID using StratifiedGroupKFold to avoid leakage.

Due to the dominance of negative statements in the dataset we made a wrapper of the Trainer class called WeightedTrainer. This wrapper includes a custom compute_loss function that performs cross-entropy loss while taking into account the modified weight of each class, which is calculated as (total labels) / (number of classes x count of this class label).

Otherwise, the procedure is fairly standard: tokenize the datasets, feed them into the trainer, train, and evaluate.

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> <!-- #### Preprocessing [optional] [More Information Needed] -->

Training Hyperparameters

training_args = TrainingArguments(
    output_dir=str(MODEL_DIR / "checkpoints"),
    num_train_epochs=20,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    learning_rate=2e-5,
    weight_decay=0.01,
    warmup_ratio=0.1,
    fp16=True,
    seed=42,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
)

<!-- #### Speeds, Sizes, Times [optional] <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed] -->

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

https://huggingface.co/datasets/hunter-lab/ignorance-classifier-testing-data

<!-- #### Factors <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed] -->

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

Precision, recall, and macro f1 were used in evaluation, as they are standard for testing the performance of a binary classifier.

[More Information Needed]

Results

{'eval_loss': 0.26312151551246643, 'eval_f1': 0.9351319475158484, 'eval_precision_class_1': 0.9629629629629629, 'eval_recall_class_1': 0.8666666666666667}