roberta-base-priority-type-generator-28k

Model Details

This model is designed to classify customer service inquiries into four priorites: Low, Medium, High and Very High.

Training Data

The model was trained on a balanced dataset of 28000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt

Training Procedure

The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.

Model Performance

The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:

• Confusion Matrix Analysis

  

  • High Priority: Most high-priority cases are correctly classified (830), with a relatively small number misclassified as Medium (10) or Very High (32). This indicates the model is quite effective at identifying High priority inquiries.
  • Low Priority: For low priority, we see significant misclassification. While 160 are correctly identified, 80 are confused as High and 86 as Medium. The model seems to struggle with distinguishing Low from Medium priority.
  • Medium Priority: The model identifies 550 cases correctly as Medium priority. However, there's noticeable confusion with High (71) and Very High (25) priorities, indicating an area for improvement.
  • Very High Priority: For Very High priority, 220 cases are correctly classified, but there's a substantial number of instances misclassified as High (4) and Medium (46), suggesting that the model finds it difficult to distinguish between Very High and the other priorities.

• Learning Curve Analysis

  

  • Training Loss: The training loss starts at approximately 1.00 and shows a steep decline, ending near 0.82. This indicates that the model is learning effectively from the training data.
  • Validation Loss: The validation loss starts just below 0.90, decreases for the first epoch, then slightly increases, ending around 0.88. This slight increase in the validation loss after an initial drop might suggest the beginnings of overfitting, where the model starts to learn specifics about the training data that do not generalize to unseen data.

• Interprating Model's Output:

  • LABEL_0 stands for High
  • LABEL_1 stands for Low
  • LABEL_2 stands for Medium
  • LABEL_3 stands for Very High