bert-base-uncased-squadv1.1-sparse-80-1x4-block-pruneofa

Model Details: 80% 1x4 Block Sparse BERT-Base (uncased) Fine Tuned on SQuADv1.1

This model has been fine-tuned for the NLP task of question answering, trained on the SQuAD 1.1 dataset. It is a result of fine-tuning a Prune Once For All 80% 1x4 block sparse pre-trained BERT-Base model, combined with knowledge distillation.

We present a new method for training sparse pre-trained Transformer language models by integrating weight pruning and model distillation. These sparse pre-trained models can be used to transfer learning for a wide range of tasks while maintaining their sparsity pattern. We show how the compressed sparse pre-trained models we trained transfer their knowledge to five different downstream natural language tasks with minimal accuracy loss.

Visualization of Prunce Once for All method from Zafrir et al. (2021). More details can be found in their paper.

How to use

Here is how to import this model in Python:

import transformers
import model_compression_research as model_comp

model = transformers.AutoModelForQuestionAnswering.from_pretrained('Intel/bert-base-uncased-squadv1.1-sparse-80-1x4-block-pruneofa')

scheduler = mcr.pruning_scheduler_factory(model, '../../examples/transformers/question-answering/config/lock_config.json')

# Train your model...

scheduler.remove_pruning()

For more code examples, refer to the GitHub Repo.

Metrics (Model Performance):

All the results are the mean of two seperate experiments with the same hyper-parameters and different seeds.

BibTeX entry and citation info

@article{zafrir2021prune,
  title={Prune Once for All: Sparse Pre-Trained Language Models},
  author={Zafrir, Ofir and Larey, Ariel and Boudoukh, Guy and Shen, Haihao and Wasserblat, Moshe},
  journal={arXiv preprint arXiv:2111.05754},
  year={2021}
}