roman-urdu-sentiment

<div align="center">

🇵🇰 Roman Urdu Sentiment Analysis

Fine-tuned XLM-RoBERTa for Roman Urdu Student Feedback

Classifies Roman Urdu text into Positive · Neutral · Negative

</div>

---

📌 Model Description

This model is a fine-tuned version of xlm-roberta-base specifically trained for sentiment analysis of Roman Urdu text — the Latin-script transliteration of Urdu widely used in Pakistani social media, messaging, and informal writing.

The model was trained on a real-world dataset of student feedback collected from Pakistani educational institutions, covering opinions on teachers, courses, classroom environment, and academic experiences. It is designed to be robust to:

• Highly variable Roman Urdu spelling (e.g. acha / accha / achha / achi)
• Code-mixed sentences with occasional English words
• Informal, noisy, social-media-style writing
• Short, context-sparse feedback phrases

Why XLM-RoBERTa?

XLM-RoBERTa Base was selected over alternatives for the following reasons:

---

🏷️ Labels

---

🚀 Quick Start

Using the pipeline API (Recommended)

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="tahamueed23/roman-urdu-sentiment",
    tokenizer="tahamueed23/roman-urdu-sentiment",
)

# Single sentence
result = classifier("ye lecture bohat acha tha")
print(result)
# [{'label': 'POSITIVE', 'score': 0.9412}]

# Batch prediction
sentences = [
    "ye lecture bohat acha tha",         # very good lecture
    "sir bilkul samjha nahi sakay",       # teacher couldn't explain at all
    "class theek thi, koi khas baat nahi" # class was okay, nothing special
]
results = classifier(sentences)
for text, res in zip(sentences, results):
    print(f"{text:<50} → {res['label']} ({res['score']:.2%})")

Using Model + Tokenizer Directly

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "tahamueed23/roman-urdu-sentiment"
tokenizer  = AutoTokenizer.from_pretrained(model_name)
model      = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

def predict_sentiment(text: str) -> dict:
    inputs = tokenizer(
        text,
        return_tensors="pt",
        truncation=True,
        max_length=128,
        padding=True,
    )
    with torch.no_grad():
        outputs = model(**inputs)

    probs   = torch.softmax(outputs.logits, dim=-1)[0]
    pred_id = int(probs.argmax())
    labels  = {0: "NEGATIVE", 1: "NEUTRAL", 2: "POSITIVE"}

    return {
        "sentiment":     labels[pred_id],
        "confidence":    round(float(probs[pred_id]), 4),
        "probabilities": {labels[i]: round(float(p), 4) for i, p in enumerate(probs)},
    }

# Example
print(predict_sentiment("zabardast teacher hai, bohat kuch seekha"))
# {
#   "sentiment": "POSITIVE",
#   "confidence": 0.9631,
#   "probabilities": {"NEGATIVE": 0.0142, "NEUTRAL": 0.0227, "POSITIVE": 0.9631}
# }

---

📊 Evaluation Results

Results on the held-out test set (10% stratified split, never seen during training).

Per-Class Metrics

Note: Neutral is the hardest class due to its inherent ambiguity in Roman Urdu — a known challenge in low-resource sentiment analysis.

---

📁 Training Data

Dataset Overview

Class Distribution (after filtering)

Dataset Quality Pipeline

The raw dataset underwent a multi-stage quality enhancement pipeline before training:

1. Deduplication — 4,089 exact duplicates removed
2. Near-duplicate flagging — similar texts filtered to prevent data leakage
3. Label confidence filtering — rows with Low confidence scores excluded
4. Quality score filtering — samples scoring < 3/7 removed (gibberish, single words)
5. Language isolation — only Roman Urdu rows retained for this model

Roman Urdu Normalization

A custom normalization dictionary was applied to unify spelling variants — a critical step for Roman Urdu which has no official orthography:

---

⚙️ Training Configuration

base_model       = "xlm-roberta-base"
max_seq_length   = 128
batch_size       = 16
gradient_accum   = 2          # effective batch = 32
epochs           = 5          # early stopping patience = 2
learning_rate    = 2e-5
lr_scheduler     = "cosine"
warmup_ratio     = 0.1
weight_decay     = 0.01
fp16             = True       # mixed precision on GPU
loss_function    = "CrossEntropyLoss (class-weighted)"
split            = "80% train / 10% val / 10% test (stratified)"
seed             = 42

Class-Weighted Loss

To handle class imbalance (Positive >> Neutral), training used sklearn.utils.class_weight.compute_class_weight('balanced') to assign higher loss penalties for the minority Neutral class. This significantly improves recall on the Neutral class without sacrificing Positive/Negative performance.

Early Stopping

Training used EarlyStoppingCallback(patience=2) monitoring eval_f1_macro. The best checkpoint is automatically restored at the end of training.

---

🛠️ Training Environment

---

⚠️ Limitations & Biases

• Domain-specific: Trained on student feedback; may underperform on social media, product reviews, or political text
• Informal Roman Urdu only: Does not support Urdu script (use a dedicated Urdu model for that)
• Spelling variation: Despite normalization, very unusual spellings not in the training vocabulary may be misclassified
• Sarcasm & irony: The model does not reliably detect sarcasm — a known hard problem in Roman Urdu NLP
• Short texts: Texts under 3 words may lack sufficient context for accurate prediction
• Regional dialect: May reflect biases from Pakistani student population data

---

🔬 Example Predictions

---

📚 Citation

If you use this model in your research or project, please cite:

@misc{mueed2025romanurdusenti,
  author       = {Taha Mueed},
  title        = {Roman Urdu Sentiment Analysis: Fine-tuned XLM-RoBERTa on Student Feedback},
  year         = {2026},
  publisher    = {HuggingFace},
  journal      = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/tahamueed23/roman-urdu-sentiment}},
}

---

👤 About the Author

Taha Mueed

NLP Researcher | Low-Resource Language Specialist | Pakistani Language AI

• 🔗 HuggingFace: @tahamueed23

---

📄 License

This model is released under the MIT License. You are free to use, modify, and distribute it for both research and commercial purposes with attribution.

---

<div align="center">

Built with ❤️ for the Roman Urdu NLP community

If this model helped your research, please ⭐ star the repository!

</div>