customer-support-ai

Customer Support AI

A machine learning model for classifying banking customer-support messages into their corresponding customer intent.

This project demonstrates a complete machine-learning workflow, including dataset inspection, data cleaning, train/validation/test splitting, model training, evaluation, error analysis, and prediction.

Project Overview

Customer-support systems receive a large number of messages every day. Automatically identifying the intent behind each message can help route customer queries to the correct support workflow.

This model takes a customer's message as input and predicts one of 77 banking-related customer-support intents.

Example

Input:

My card hasn't arrived yet

Prediction:

card_arrival

Another example:

Input:

I want to cancel my transfer

Prediction:

cancel_transfer

---

Task

Text Classification / Customer Intent Classification

The model performs multi-class classification on customer-support messages.

Input

A natural-language customer-support message.

Output

One of 77 predefined banking customer-support intents.

---

Dataset

The model was trained using the Banking77 dataset.

Banking77 contains banking-related customer queries categorized into 77 different intents.

The dataset was processed through the following pipeline:

Raw Dataset
     ↓
Dataset Inspection
     ↓
Data Cleaning
     ↓
Train / Validation / Test Split
     ↓
Model Training
     ↓
Validation Evaluation
     ↓
Error Analysis
     ↓
Final Test Evaluation

---

Model

The trained model is a scikit-learn text-classification model saved using joblib.

Model file:

customer_support_model.pkl

The model can be loaded in Python using:

import joblib

model = joblib.load("customer_support_model.pkl")

prediction = model.predict([
    "My card hasn't arrived yet"
])

print(prediction)

Expected output:

['card_arrival']

---

Model Performance

The model was evaluated using separate validation and test datasets.

Validation Results

Validation records:

1,000

Validation accuracy:

83.90%

Final Test Results

Test records:

1,000

Test accuracy:

84.70%

Additional test metrics:

The final test evaluation was performed on data that was not used during model training.

---

Error Analysis

Error analysis was performed on the validation dataset to understand where the model makes incorrect predictions.

Validation Results

Validation records: 1,000
Correct predictions: 839
Incorrect predictions: 161
Error rate: 16.10%

The analysis showed that some errors occur between semantically similar customer-support intents.

For example, messages involving:

• pending payments
• failed payments
• reversed payments
• top-up problems
• card-related problems
• transfer-related problems

can sometimes contain similar language, making them harder to classify.

The project includes an error-analysis script that generates:

reports/validation_predictions.csv

---

Supported Intent Categories

The model supports 77 banking customer-support intents, including:

Refund_not_showing_up
activate_my_card
age_limit
apple_pay_or_google_pay
atm_support
automatic_top_up
balance_not_updated_after_bank_transfer
balance_not_updated_after_cheque_or_cash_deposit
beneficiary_not_allowed
cancel_transfer
card_about_to_expire
card_acceptance
card_arrival
card_delivery_estimate
card_linking
card_not_working
card_payment_fee_charged
card_payment_not_recognised
card_payment_wrong_exchange_rate
card_swallowed
cash_withdrawal_charge
cash_withdrawal_not_recognised
change_pin
compromised_card
contactless_not_working
country_support
declined_card_payment
declined_cash_withdrawal
declined_transfer
direct_debit_payment_not_recognised
disposable_card_limits
edit_personal_details
exchange_charge
exchange_rate
exchange_via_app
extra_charge_on_statement
failed_transfer
fiat_currency_support
get_disposable_virtual_card
get_physical_card
getting_spare_card
getting_virtual_card
lost_or_stolen_card
lost_or_stolen_phone
order_physical_card
passcode_forgotten
pending_card_payment
pending_cash_withdrawal
pending_top_up
pending_transfer
pin_blocked
receiving_money
request_refund
reverted_card_payment?
supported_cards_and_currencies
terminate_account
top_up_by_bank_transfer_charge
top_up_by_card_charge
top_up_by_cash_or_cheque
top_up_failed
top_up_limits
top_up_reverted
topping_up_by_card
transaction_charged_twice
transfer_fee_charged
transfer_into_account
transfer_not_received_by_recipient
transfer_timing
unable_to_verify_identity
verify_my_identity
verify_source_of_funds
verify_top_up
virtual_card_not_working
visa_or_mastercard
why_verify_identity
wrong_amount_of_cash_received
wrong_exchange_rate_for_cash_withdrawal

---

Project Features

The complete project contains:

• Dataset inspection
• Data cleaning
• Train/validation/test splitting
• Machine-learning model training
• Validation evaluation
• Classification report
• Error analysis
• Final test evaluation
• Interactive prediction
• Streamlit web application
• Model serialization using Joblib

---

Project Structure

customer-support-annotation/
│
├── annotation/
│
├── data/
│   ├── raw/
│   │   └── train.csv
│   │
│   └── processed/
│       ├── cleaned_train.csv
│       ├── train.csv
│       ├── validation.csv
│       └── test.csv
│
├── models/
│   └── customer_support_model.pkl
│
├── reports/
│   └── validation_predictions.csv
│
├── scripts/
│   ├── clean_dataset.py
│   ├── error_analysis.py
│   ├── evaluate_model.py
│   ├── inspect_dataset.py
│   ├── predict.py
│   ├── split_dataset.py
│   ├── test_model.py
│   └── train_model.py
│
├── app.py
├── requirements.txt
└── README.md

---

Interactive Prediction

The project includes an interactive prediction script.

Run:

python scripts/predict.py

Example:

===== CUSTOMER SUPPORT AI =====
Type 'exit' to stop.

Customer message: My card hasn't arrived yet
Predicted Intent: card_arrival

---

Live Demo

The model is also integrated into a web application that allows users to enter customer-support messages and receive predicted intents.

Live Demo:

Add your Streamlit application URL here.

[https://customer-support-ai-gkmuxmcdqfjcwk3q6tpidm.streamlit.app/]

---

Installation

Clone the project:

git clone https://github.com/princechouhan3/customer-support-ai.git
cd customer-support-annotation

Install dependencies:

pip install -r requirements.txt

Run the prediction application:

python scripts/predict.py

---

Source Code

The complete source code, training scripts, evaluation scripts, dataset-processing pipeline, and application code are available on GitHub.

GitHub Repository:

https://github.com/princechouhan3/customer-support-ai

---

Potential Applications

This type of intent-classification model can be used as a component of:

• Banking customer-support systems
• Customer-service chatbots
• Automated ticket routing
• Support ticket classification
• Customer-support analytics
• FAQ and help-desk automation
• Intent detection systems

The model itself is a classification component and can be integrated into a larger customer-support or chatbot system.

---

Limitations

This model was trained and evaluated using the Banking77 dataset.

Its performance on real-world customer messages may differ from the reported test performance.

The model should not be used as a production banking decision system without additional:

• domain-specific validation
• security testing
• monitoring
• bias evaluation
• robustness testing
• production testing
• human oversight where required

The model is intended primarily as a machine-learning project and portfolio demonstration.

---

Future Improvements

Possible future improvements include:

• Improving classification accuracy
• Performing hyperparameter tuning
• Adding confidence scores
• Handling unknown or out-of-domain queries
• Improving error handling
• Adding more real-world customer-support data
• Comparing multiple ML algorithms
• Deploying the model as an API
• Integrating the classifier with a conversational AI system
• Adding monitoring and model evaluation pipelines

---

Project

Customer Support AI — Banking Intent Classification

This project demonstrates an end-to-end machine-learning workflow from raw customer-support data to a trained and evaluated classification model and an interactive prediction application.

---

License

This project is released under the MIT License.