roberta-base-category-type-generator-53k

Model Details

This model is designed to classify customer service inquiries into five categories: Technical Issues, Billing & Payment, Product Inquiries, Account Management, and Policy Questions.

Training Data

The model was trained on a balanced dataset of 53000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt

Training Procedure

The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.

Model Performance

The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:

• Confusion Matrix Analysis

  

  • Technical Issues: Some misclassification noted, particularly with Policy Questions.
  • Billing & Payment: High accuracy with some instances misclassified as Technical Issues.
  • Product Inquiries: Very high accuracy with few misclassifications.
  • Account Management: Notable confusion with Policy Questions.
  • Policy Questions: Highest accuracy among all categories.

• Learning Curve Analysis

  

  • The training loss decreased consistently, indicating good model learning.
  • The validation loss remained close to the training loss and did not show signs of increasing, suggesting that the model was generalizing well without overfitting.

• Interprating Model's Output:

  • LABEL_0 stands for Account Management
  • LABEL_1 stands for Billing & Payment
  • LABEL_2 stands for Policy Questions
  • LABEL_3 stands for Product Inquiries
  • LABEL_4 stands for Technical Issues