ruroberta-ecom-tech-best

Food Delivery Feedback Multi-Label Classification Model

This model was developed for multi-label classification of customer feedback in the food delivery domain. It can identify up to 50 different aspects/issues from user reviews and feedback messages.

Model Description

The model is designed to analyze customer feedback on Russian and automatically categorize it into multiple relevant labels, helping businesses quickly identify and address various aspects of their food delivery service.

Key Features:

• Multi-label classification across 50 distinct categories
• Specialized for food delivery domain feedback
• Handles various aspects including delivery speed, food quality, packaging, customer service, etc.
• Supports Russian language feedback processing

Use Cases:

• Automated feedback categorization
• Customer satisfaction analysis
• Service quality monitoring
• Issue identification and prioritization
• Operational improvement insights

Performance:

• Trained on about 3000 customer feedback messages
• Achieves Full Match Accuracy on test set
• Optimized for real-world food delivery feedback scenarios

Evaluation Metric: Full Match Accuracy

The model is evaluated using a strict full match accuracy metric that requires exact matching between predicted and target labels.

Metric Description:

• Type: Exact Match / Perfect Match Accuracy
• Range: 0 to 1 (or 0% to 100%)
• Format: Labels are represented as space-separated indices (e.g., "0 1 10")

Example:

# Correct prediction (100% accuracy):
Target:     "0 1 10"
Prediction: "0 1 10"

# Incorrect predictions (0% accuracy):
Target:     "0 1 10"
Prediction: "0 1"      # Partial match is considered incorrect
Target:     "0 1 10"
Prediction: "0 1 10 2" # Extra label is considered incorrect

Labels Include:

• Delivery time issues
• Food quality concerns
• Order accuracy
• Packaging condition
• Courier behavior
• App/website functionality
• Payment issues etc.

-> Full labels see here

Usage

The model can be used for:

• Real-time feedback classification
• Batch processing of historical feedback
• Customer support automation
• Service quality analytics

Training

The model was trained as part of an NLP competition focused on improving customer feedback analysis in the food delivery industry. It uses state-of-the-art transformer architecture optimized for multi-label classification tasks.

Limitations

• Primarily optimized for Russian language feedback
• May require fine-tuning for specific regional contexts
• Best suited for food delivery domain specifically

How to use with PyTorch and Transfomers

import torch
from transformers import RobertaTokenizer, RobertaForSequenceClassification


model_name = 'ai-forever/ruRoberta-large'
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

tokenizer = RobertaTokenizer.from_pretrained(model_name)
class_descriptions = pd.read_csv('files/trends_description.csv')

class BERTClass(torch.nn.Module):
    def __init__(self):
        super(BERTClass, self).__init__()
        self.bert_model = RobertaForSequenceClassification.from_pretrained(
            'metanovus/ruroberta-ecom-tech-best',
            return_dict=True,
            problem_type='multi_label_classification',
            num_labels=50
        )

    def forward(self, input_ids, attn_mask, token_type_ids):
        output = self.bert_model(
            input_ids,
            attention_mask=attn_mask,
            token_type_ids=token_type_ids
        )
        return output.logits

model = BERTClass().to(device)