roberta-base-service-type-generator-28k

Model Details

This model is designed to classify customer service inquiries into nine services: SAP ERP, Atlassian, Adobe, Salesforce, Reporting, Microsoft Power Platform, Microsoft SharePoint, Snowflake, Microsoft Office.

Training Data

The model was trained on a balanced dataset of 28000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt

Training Procedure

The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.

Model Performance

The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:

• Confusion Matrix Analysis

  

  • SAP ERP: Most instances are classified correctly (400), with a few misclassifications as Adobe (1) and Microsoft Office (1).
  • Atlassian: This category has perfect classification with all instances (410) correctly identified.
  • Adobe: Also has high accuracy with 390 instances correctly classified and a single misclassification as SAP ERP (1).
  • Salesforce: There are 450 correctly classified instances, but there's some confusion with Reporting (16) and Microsoft Power Platform (1), and Microsoft SharePoint (1).
  • Reporting: There are 59 correct predictions. However, a significant number of Reporting instances are misclassified as Salesforce (36).
  • Microsoft Power Platform: This category has 320 correct classifications, with a few instances misclassified as Reporting (5), Snowflake (2), and Microsoft SharePoint (2).
  • Microsoft SharePoint: Most instances are correctly classified (390), with minimal confusion with other services.
  • Snowflake: There are 300 instances correctly identified, with a single instance misclassified as Microsoft Power Platform (1).
  • Microsoft Office: This category has 30 instances, all correctly classified with no misclassifications.

• Learning Curve Analysis

  

  • Training Loss: The training loss starts at approximately 0.12 and decreases to about 0.065. The steady decline indicates the model is learning effectively from the training data.
  • Validation Loss: The validation loss starts around 0.11, decreases, then increases slightly at epoch 2 before declining again, ending around 0.075. This pattern suggests some variation in model performance on the validation set, but overall, the validation loss follows a downward trend, indicating improving model generalization.

• Interprating Model's Output:

  • LABEL_0 stands for Adobe
  • LABEL_1 stands for Atlassian
  • LABEL_2 stands for Microsoft Office
  • LABEL_3 stands for Microsoft Power Platform
  • LABEL_4 stands for Microsoft SharePoint
  • LABEL_5 stands for Reporting
  • LABEL_6 stands for SAP ERP
  • LABEL_7 stands for Salesforce
  • LABEL_8 stands for Snowflake