code-switching-codesaviours-si26-MuhammadHassaan

Roman Urdu-English Language Identification Model

Project Information

Project: Code Saviours SI-26 — Week 7
Intern: Muhammad Hassaan
Student ID: SI26-ML-MH-004
Department: AI/ML

Model Description

This model is a fine-tuned XLM-RoBERTa model for word-level language identification in Roman Urdu-English code-switched text.

The model predicts one of three labels for each word:

• URD — Roman Urdu
• ENG — English
• MIX — Mixed Roman Urdu-English token

Base Model

The model was fine-tuned from:

xlm-roberta-base

Dataset

The original dataset contained 150 Roman Urdu-English code-switched sentences.

During Week 7, additional MIX examples were added because the original dataset contained very few MIX-labelled tokens.

The augmented dataset contained 180 sentences.

The dataset was divided into:

• 144 training sentences
• 36 testing sentences

Training

XLM-RoBERTa was fine-tuned for token classification using GPU training.

Training configuration included:

• Epochs: 5
• Learning rate: 2e-5
• Training batch size: 16
• Evaluation strategy: Every epoch
• Labels: URD, ENG, MIX

Because the MIX class was underrepresented, sentences containing MIX-labelled tokens were oversampled during the final training experiment.

Evaluation

The model was evaluated using per-class F1 scores for:

• URD
• ENG
• MIX

The final balanced model performed better on the MIX class than the original training experiment.

Important: The MIX test set contained only a small number of MIX-labelled examples, so the MIX F1 score should be interpreted carefully.

Final F1 Scores

URD F1 : 0.9865 ENG F1 : 0.9853 MIX F1 : 1.0000 Macro F1 : 0.9906 Accuracy : 0.9863

Example

Input:

Aaj mera meeting schedule bohot busy hai

Expected language identification:

• Aaj → URD
• mera → URD
• meeting → ENG
• schedule → ENG
• bohot → URD
• busy → ENG
• hai → URD

Mixed-token example:

Mujhe fileupload karni hai

• Mujhe → URD
• fileupload → MIX
• karni → URD
• hai → URD

Limitations

• The dataset is relatively small.
• The MIX class has fewer examples than URD and ENG.
• Roman Urdu spelling is highly variable.
• Performance may decrease on unseen spellings or sentence styles.
• The model should be considered a research and internship project rather than a production-ready language identification system.

Future Improvements

Future work could include:

• Collecting a larger Roman Urdu-English dataset.
• Adding more naturally occurring MIX examples.
• Improving class balance.
• Testing on a larger unseen dataset.
• Performing additional hyperparameter tuning.

Author

Muhammad Hassaan
Student ID: SI26-ML-MH-004
Code Saviours SI-26