Downloads · 30 days
103
1% of all-time downloads
canwenxu/BERT-of-Theseus-MNLI
BERT-of-Theseus-MNLI is a feature extraction model from canwenxu. Use it when you need embeddings to search or compare text. It is set up for transformers.
See our paper "BERT-of-Theseus: Compressing BERT by Progressive Module Replacing".
Downloads · 30 days
103
1% of all-time downloads
All-time downloads
7.9K
Public
Repo size
804 MB
Likes
1
Public
Click a slice to open those files.
.bin268 MB · 50%
From the Hugging Face model README
See our paper "BERT-of-Theseus: Compressing BERT by Progressive Module Replacing".
BERT-of-Theseus is a new compressed BERT by progressively replacing the components of the original BERT.

We provide a 6-layer pretrained model on MNLI as a general-purpose model, which can transfer to other sentence classification tasks, outperforming DistillBERT (with the same 6-layer structure) on six tasks of GLUE (dev set).
| Method | MNLI | MRPC | QNLI | QQP | RTE | SST-2 | STS-B |
|---|---|---|---|---|---|---|---|
| BERT-base | 83.5 | 89.5 | 91.2 | 89.8 | 71.1 | 91.5 | 88.9 |
| DistillBERT | 79.0 | 87.5 | 85.3 | 84.9 | 59.9 | 90.7 | 81.2 |
| BERT-of-Theseus | 82.1 | 87.5 | 88.8 | 88.8 | 70.1 | 91.8 | 87.8 |
Please Note: this checkpoint is for Intermediate-Task Transfer Learning so it does not include the classification head for MNLI! Please fine-tune it before use (like DistilBERT).