minilm-finetuned-emotion

<!-- This model card has been generated automatically according to the information Keras had access to. You should probably proofread and complete it, then remove this comment. -->

laxsvips/minilm-finetuned-emotion

This model is a fine-tuned version of microsoft/MiniLM-L12-H384-uncased on the Hugging Face emotion (https://huggingface.co/datasets/emotion) dataset. It achieves the following results on the evaluation set:

• Train Loss: 0.1168
• Train Accuracy: 0.9446
• Validation Loss: 0.1709
• Validation Accuracy: 0.9350
• Epoch: 4

Model description

MiniLM: Small and Fast Pre-trained Models for Language Understanding and Generation

MiniLM is a distilled model from the paper "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers".

Intended uses & limitations

This model has been created as a learning guide on:

• How to implement a text classification model using Hugging Face Transformers in TensorFlow
• How to handle imbalanced class distribution

How to use the model

from transformers import pipeline

model_cpt = "laxsvips/minilm-finetuned-emotion"
pipe = pipeline("text-classification", model=model_cpt)
predicted_scores = pipe("I am so glad you could help me")
print(predicted_scores)

The results:

[[{'label': 'sadness', 'score': 0.003758953418582678},
  {'label': 'joy', 'score': 0.9874302744865417},
  {'label': 'love', 'score': 0.00610917154699564},
  {'label': 'anger', 'score': 9.696640336187556e-05},
  {'label': 'fear', 'score': 0.0006420552381314337},
  {'label': 'surprise', 'score': 0.00196251692250371}]]

Training and evaluation data

Emotion

Emotion is a dataset of English Twitter messages with six basic emotions: anger, fear, joy, love, sadness, and surprise.

Training procedure

Refer to the Colab notebook

Training hyperparameters

The following hyperparameters were used during training:

• optimizer: 'Adam',
• learning_rate': 5e-05,
• batch_size : 64
• num_epochs - 5

Training results

Evaluation Metrics

{'accuracy': 0.935,
 'precision': 0.937365614416424,
 'recall': 0.935,
 'f1_score': 0.9355424419858925}

Framework versions

• Transformers 4.26.1
• TensorFlow 2.11.0
• Datasets 2.9.0
• Tokenizers 0.13.2

References

1. https://www.youtube.com/watch?v=u--UVvH-LIQ
2. https://huggingface.co/docs/transformers
3. https://www.tensorflow.org/api_docs/python/tf