code-switching-codesaviours-si26-zainab

Roman Urdu-English Code-Switching Language Identifier

Token classification model that identifies whether each word in a Roman Urdu/English code-switched sentence is Urdu (URD), English (ENG), or a fused hybrid word (MIX).

Model Details

• Base model: xlm-roberta-base
• Task: Token Classification (3-class: URD / ENG / MIX)
• Fine-tuned by: Zainab Binte Khalid, Code Saviours (SI-26)
• Training data: code-switching-codesaviours-si26-zainab dataset — 1,901 real, hand-collected and hand-labeled Roman Urdu/English sentences (WhatsApp messages and personally-reviewed online comments)

Training

• 8 epochs, learning rate 2e-5, warmup ratio 0.1, weight decay 0.01
• 80/20 train/test split (1,520 / 381 sentences)
• GPU (Colab T4)

Evaluation Results

Overall F1 (URD/ENG): 0.957 (95.73%)

Note on MIX: MIX applies only to single tokens that fuse Urdu and English within one word (e.g. "loadshedding", "tension-wession") — not to sentences that simply contain both languages. This pattern is genuinely rare in natural Roman Urdu/English text; the training data contains only 2 MIX-labeled tokens out of 21,368 total, which is insufficient for the model to learn this class. This reflects the rarity of the phenomenon itself rather than a data collection gap.

Intended Use

Identifying language at the word level in Roman Urdu/English code-switched text — useful for downstream NLP tasks (sentiment analysis, translation, spell-checking) that need language-aware preprocessing for this common but under-resourced language-mixing pattern.

Limitations

• Trained on informal, conversational text (WhatsApp, social comments) — may not generalize well to formal or domain-specific text (news, academic writing)
• MIX class is effectively unlearned due to extreme rarity in training data
• No standardized Roman Urdu spelling exists, so performance may vary on unusual spelling variants not seen in training