distilbert-base-uncased-sparse-85-unstructured-pruneofa

Model Details: 85% Sparse DistilBERT-Base (uncased) Prune Once for All

This model is a sparse pre-trained model that can be fine-tuned for a wide range of language tasks. The process of weight pruning is forcing some of the weights of the neural network to zero. Setting some of the weights to zero results in sparser matrices. Updating neural network weights does involve matrix multiplication, and if we can keep the matrices sparse while retaining enough important information, we can reduce the overall computational overhead. The term "sparse" in the title of the model indicates a ratio of sparsity in the weights; for more details, you can read Zafrir et al. (2021).

Visualization of Prunce Once for All method from Zafrir et al. (2021):

How to use

Here is an example of how to import this model in Python:

import transformers

model = transformers.AutoModelForQuestionAnswering.from_pretrained('Intel/distilbert-base-uncased-sparse-85-unstructured-pruneofa')

For more code examples, refer to the GitHub Repo.

Metrics (Model Performance):

All the results are the mean of two seperate experiments with the same hyper-parameters and different seeds.

BibTeX entry and citation info

@article{zafrir2021prune,
  title={Prune Once for All: Sparse Pre-Trained Language Models},
  author={Zafrir, Ofir and Larey, Ariel and Boudoukh, Guy and Shen, Haihao and Wasserblat, Moshe},
  journal={arXiv preprint arXiv:2111.05754},
  year={2021}
}