roberta-base-category-type-generator-43k

Model Details

This model is designed to classify customer service inquiries into five categories: Technical Issues, Billing & Payment, Product Inquiries, Account Management, and Policy Questions.

Training Data

The model was trained on a balanced dataset of 43000 entries composed of anonymized customer service inquiries. Each category contained a similar number of examples to prevent class imbalance. https://github.com/amosproj/amos2023ws01-ticket-chat-ai/tree/main/Backend/app/model/test_data/test_data_with_gpt

Training Procedure

The model was fine-tuned over four epochs for a sequence classification task. We utilized a batch size of 4 and an Adam optimizer with a learning rate of 2e-5.

Model Performance

The model's performance was evaluated using a confusion matrix and a learning curve, as detailed below:

• Confusion Matrix Analysis

  

• Learning Curve Analysis

  

  • Training Loss: The training loss starts at approximately 0.42 and decreases steadily to around 0.32. This is a good sign as it suggests that the model is learning and improving its prediction on the training data with each epoch.
  • Validation Loss: The validation loss starts around 0.38 and decreases slightly after the first epoch but then flattens out and remains almost constant around 0.36. The flattening of the validation loss indicates that further learning improvements are marginal, and the model is not gaining additional predictive power from further training on this dataset.