Downloads · 30 days
13
2% of all-time downloads
TinyBERT_L-4_H-312_v2-distill-AllNLI
ceshine/TinyBERT_L-4_H-312_v2-distill-AllNLI
TinyBERT_L-4_H-312_v2-distill-AllNLI is a feature extraction model from ceshine. Use it when you need embeddings to search or compare text. It is set up for transformers.
This is distilled from the bert-base-nli-stsb-mean-tokens pre-trained model from Sentence-Transformers.
All-time downloads
633
Public
Repo size
230 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin57.4 MB · 50%
At a glance
- Task
- Feature Extraction
- Library
- transformers
- Model type
- bert
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- May 19, 2021
- SHA
- 57883374
Model card
From the Hugging Face model README
TinyBERT_L-4_H-312_v2 English Sentence Encoder
This is distilled from the bert-base-nli-stsb-mean-tokens pre-trained model from Sentence-Transformers.
The embedding vector is obtained by mean/average pooling of the last layer's hidden states.
Update 20210325: Added the attention matrices imitation objective as in the TinyBERT paper, and the distill target has been changed from distilbert-base-nli-stsb-mean-tokens to bert-base-nli-stsb-mean-tokens (they have almost the same STSb performance).
Model Comparison
We compute cosine similarity scores of the embeddings of the sentence pair to get the spearman correlation on the STS benchmark (bigger is better):
| Dev | Test | |
|---|---|---|
| bert-base-nli-stsb-mean-tokens | .8704 | .8505 |
| distilbert-base-nli-stsb-mean-tokens | .8667 | .8516 |
| TinyBERT_L-4_H-312_v2-distill-AllNLI | .8587 | .8283 |
| TinyBERT_L-4_H (20210325) | .8551 | .8341 |
- Task
- Feature Extraction
- Library
- transformers
- Type
- bert
- Created
- Mar 2, 2022
- Updated
- May 19, 2021