Downloads · 30 days
22
0% of all-time downloads
madlag/bert-base-uncased-squadv1-x2.01-f89.2-d30-hybrid-rewind-opt-v1
bert-base-uncased-squadv1-x2.01-f89.2-d30-hybrid-rewind-opt-v1 is a question answering model from madlag. Use it when the input is a question plus a passage. It is set up for transformers. The card lists the license as mit.
This model was created using the nnpruning python library: the linear layers contains 30.0% of the original weights.
Downloads · 30 days
22
0% of all-time downloads
All-time downloads
9K
Public
Repo size
1.2 GB
Likes
0
Public
Click a slice to open those files.
.h5436 MB · 52%
From the Hugging Face model README
This model was created using the nn_pruning python library: the linear layers contains 30.0% of the original weights.
This model CANNOT be used without using nn_pruning optimize_model function, as it uses NoNorms instead of LayerNorms and this is not currently supported by the Transformers library.
It uses ReLUs instead of GeLUs as in the initial BERT network, to speedup inference.
This does not need special handling, as it is supported by the Transformers library, and flagged in the model config by the "hidden_act": "relu" entry.
The model contains 45.0% of the original weights overall (the embeddings account for a significant part of the model, and they are not pruned by this method).
With a simple resizing of the linear matrices it ran 2.01x as fast as bert-base-uncased on the evaluation. This is possible because the pruning method lead to structured matrices: to visualize them, hover below on the plot to see the non-zero/zero parts of each matrix.
<div class="graph"><script src="/madlag/bert-base-uncased-squadv1-x2.01-f89.2-d30-hybrid-rewind-opt-v1/raw/main/model_card/density_info.js" id="c3b978cc-6d18-4fd0-a24b-e4369569d64d"></script></div>In terms of accuracy, its F1 is 89.19, compared with 88.5 for bert-base-uncased, a F1 gain of 0.69.
This model was fine-tuned from the HuggingFace model checkpoint on SQuAD1.1, and distilled from the model bert-large-uncased-whole-word-masking-finetuned-squad This model is case-insensitive: it does not make a difference between english and English.
A side-effect of the block pruning is that some of the attention heads are completely removed: 55 heads were removed on a total of 144 (38.2%). Here is a detailed view on how the remaining heads are distributed in the network after pruning.
<div class="graph"><script src="/madlag/bert-base-uncased-squadv1-x2.01-f89.2-d30-hybrid-rewind-opt-v1/raw/main/model_card/pruning_info.js" id="7de38b6d-774c-4313-a5a4-8e32f554d9ec"></script></div>| Dataset | Split | # samples |
|---|---|---|
| SQuAD1.1 | train | 90.6K |
| SQuAD1.1 | eval | 11.1k |
Python: 3.8.5
Machine specs:
Memory: 64 GiB
GPUs: 1 GeForce GTX 3090, with 24GiB memory
GPU driver: 455.23.05, CUDA: 11.1
Pytorch model file size: 374MB (original BERT: 420MB)
| Metric | # Value | # Original (Table 2) | Variation |
|---|---|---|---|
| EM | 82.21 | 80.8 | +1.41 |
| F1 | 89.19 | 88.5 | +0.69 |
Install nn_pruning: it contains the optimization script, which just pack the linear layers into smaller ones by removing empty rows/columns.
pip install nn_pruning
Then you can use the transformers library almost as usual: you just have to call optimize_model when the pipeline has loaded.
from transformers import pipeline
from nn_pruning.inference_model_patcher import optimize_model
qa_pipeline = pipeline(
"question-answering",
model="madlag/bert-base-uncased-squadv1-x2.01-f89.2-d30-hybrid-rewind-opt-v1",
tokenizer="madlag/bert-base-uncased-squadv1-x2.01-f89.2-d30-hybrid-rewind-opt-v1"
)
print("bert-base-uncased parameters: 200.0M")
print(f"Parameters count (includes only head pruning, not feed forward pruning)={int(qa_pipeline.model.num_parameters() / 1E6)}M")
qa_pipeline.model = optimize_model(qa_pipeline.model, "dense")
print(f"Parameters count after complete optimization={int(qa_pipeline.model.num_parameters() / 1E6)}M")
predictions = qa_pipeline({
'context': "Frédéric François Chopin, born Fryderyk Franciszek Chopin (1 March 1810 – 17 October 1849), was a Polish composer and virtuoso pianist of the Romantic era who wrote primarily for solo piano.",
'question': "Who is Frederic Chopin?",
})
print("Predictions", predictions)