biomedbert-hash-nano

BiomedBERT Hash Nano

This is a 970K parameter BERT encoder-only model trained on data from PubMed. The raw data was transformed using PaperETL with the results stored as a local dataset via the Hugging Face Datasets library.

biomedbert-hash-nano is built with the BERT Hash architecture as described in the following links.

• BERT Hash Nano
• Training Tiny Language Models with Token Hashing

Usage

biomedbert-hash-nano can be loaded using Hugging Face Transformers as follows. Note that given that this is a custom architecture, trust_remote_code needs to be set.

from transformers import AutoModel

model = AutoModel.from_pretrained("neuml/biomedbert-hash-nano", trust_remote_code=True)

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

Evaluation Results

This Medical Abstracts Text Classification Dataset was used to evaluate the model's performance. A handful of biomedical models and general models were selected for comparison.

Metrics were generated using Hugging Face's standard run_glue script as shown below.

python run_glue.py --model_name_or_path neuml/biomedbert-hash-nano --dataset-name medclassify --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 1e-4 --num_train_epochs 4 --output_dir outputs --trust-remote-code True

Note: The original dataset was saved locally as medclassify the the condition_label column renamed to label to work more easily with the glue script

As we can see, this model performs very well against models much larger in size. This dataset is a challenging one!

More Information

Read more about the model in this article and this paper.