Downloads · 30 days
43
5% of all-time downloads
bioformer-8L-squad1
bioformers/bioformer-8L-squad1
bioformer-8L-squad1 is a question answering model from bioformers. Use it when the input is a question plus a passage. It is set up for transformers. The card lists the license as apache-2.0.
bioformer-8L fined-tuned on the SQuAD1 dataset for 3 epochs.
All-time downloads
786
Public
Parameters
42.3M
507 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin169 MB · 50%
Parameter types
How the weights are stored.
F3242.3M · 100%
Try a prompt
Model card
From the Hugging Face model README
bioformer-8L fined-tuned on the SQuAD1 dataset for 3 epochs.
The fine-tuning process was performed on a single P100 GPUs (16GB). The hyperparameters are:
max_seq_length=512
per_device_train_batch_size=16
gradient_accumulation_steps=1
total train batch size (w. parallel, distributed & accumulation) = 16
learning_rate=3e-5
num_train_epochs=3
Evaluation results
"eval_exact_match": 78.55250709555345
"eval_f1": 85.91482799690257
Bioformer's performance is on par with DistilBERT (EM/F1: 77.7/85.8), although Bioformer was pretrained only on biomedical texts.
Speed
In our experiments, the inference speed of Bioformer is 3x as fast as BERT-base/BioBERT/PubMedBERT, and is 40% faster than DistilBERT.
- Task
- Question Answering
- Library
- transformers
- Type
- bert
- License
- apache-2.0
- Languages
- en
- Created
- Mar 2, 2022
- Updated
- Aug 2, 2023