distilbert-base-uncased-emotion

Distilbert-base-uncased-emotion

Model description:

Distilbert is created with knowledge distillation during the pre-training phase which reduces the size of a BERT model by 40%, while retaining 97% of its language understanding. It's smaller, faster than Bert and any other Bert-based model.

Distilbert-base-uncased finetuned on the emotion dataset using HuggingFace Trainer with below Hyperparameters

 learning rate 2e-5, 
 batch size 64,
 num_train_epochs=8,

Model Performance Comparision on Emotion Dataset from Twitter:

How to Use the model:

from transformers import pipeline
classifier = pipeline("text-classification",model='bhadresh-savani/distilbert-base-uncased-emotion', return_all_scores=True)
prediction = classifier("I love using transformers. The best part is wide range of support and its easy to use", )
print(prediction)

"""
Output:
[[
{'label': 'sadness', 'score': 0.0006792712374590337}, 
{'label': 'joy', 'score': 0.9959300756454468}, 
{'label': 'love', 'score': 0.0009452480007894337}, 
{'label': 'anger', 'score': 0.0018055217806249857}, 
{'label': 'fear', 'score': 0.00041110432357527316}, 
{'label': 'surprise', 'score': 0.0002288572577526793}
]]
"""

Dataset:

Twitter-Sentiment-Analysis.

Training procedure

Colab Notebook

Eval results

{
'test_accuracy': 0.938,
 'test_f1': 0.937932884041714,
 'test_loss': 0.1472451239824295,
 'test_mem_cpu_alloc_delta': 0,
 'test_mem_cpu_peaked_delta': 0,
 'test_mem_gpu_alloc_delta': 0,
 'test_mem_gpu_peaked_delta': 163454464,
 'test_runtime': 5.0164,
 'test_samples_per_second': 398.69
 }

Citation

If you use this model in your research, please cite it:

@misc{savani2021distilbertemotion,
  author    = {Savani, Bhadresh},
  title     = {{distilbert-base-uncased-emotion: DistilBERT Fine-tuned for Emotion Classification}},
  year      = {2021},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/XXXX},
  url       = {https://huggingface.co/bhadresh-savani/distilbert-base-uncased-emotion}
}

## Reference:
* [Natural Language Processing with Transformer By Lewis Tunstall, Leandro von Werra, Thomas Wolf](https://learning.oreilly.com/library/view/natural-language-processing/9781098103231/)