Downloads · 30 days
300
14% of all-time downloads
distilroberta-base-sst2-distilled
azizbarank/distilroberta-base-sst2-distilled
distilroberta-base-sst2-distilled is a text classification model from azizbarank. Use it when you need a label for a piece of text. It is set up for transformers.
Distilled version of the RoBERTa model fine-tuned on the SST-2 part of the GLUE dataset. It was obtained from the "teacher" RoBERTa model by using task-specific knowledge distillation. Since the teacher was fine-tuned…
All-time downloads
2.1K
Public
Repo size
8.9 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin329 MB · 99%
At a glance
- Task
- Text Classification
- Library
- transformers
- Model type
- roberta
- Access
- Public
- Created
- Aug 23, 2022
- Updated
- Oct 8, 2022
- SHA
- dda0cb40
Try a prompt
Model card
From the Hugging Face model README
Distilled version of the RoBERTa model fine-tuned on the SST-2 part of the GLUE dataset. It was obtained from the "teacher" RoBERTa model by using task-specific knowledge distillation. Since the teacher was fine-tuned on the SST-2, the final model as well is ready to be used in sentiment analysis tasks.
Modifications to the original RoBERTa model:
The final distilled model was able to achieve 92% accuracy on the SST-2 dataset. Given the original RoBERTa achieves 94.8% accuracy on the same dataset with much more parameters (125M) and that the distilled model is nearly twice as fast as it is, the accuracy is an impressive result.
Training Results after Hyperparameter Tuning
| Epoch | Training Loss | Validation Loss | Accuracy |
|---|---|---|---|
| 1 | 0.144000 | 0.379220 | 0.907110 |
| 2 | 0.108500 | 0.466671 | 0.911697 |
| 3 | 0.078600 | 0.359551 | 0.915138 |
| 4 | 0.057400 | 0.358214 | 0.920872 |
Usage
To use the model from the 🤗/transformers library
# !pip install transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("azizbarank/distilroberta-base-sst2-distilled")
model = AutoModelForSequenceClassification.from_pretrained("azizbarank/distilroberta-base-sst2-distilled")
- Task
- Text Classification
- Library
- transformers
- Type
- roberta
- Created
- Aug 23, 2022
- Updated
- Oct 8, 2022