urdu-roman-transliterator-tiny-aya-V3

Urdu → Roman Urdu Transliterator (Tiny Aya, QLoRA fine-tune)

This model transliterates Urdu script text into Roman Urdu (Urdu written in Latin script), fine-tuned from CohereLabs/tiny-aya-global using QLoRA.

Unlike translation, this is a transliteration task: the output preserves the original words and meaning, just rendered in Latin script instead of Urdu script.

Example:

Model Details

• Base model: CohereLabs/tiny-aya-global (3.35B parameters, Cohere2 architecture)
• Fine-tuning method: QLoRA (4-bit quantized base, LoRA adapters merged into full weights for this release)
• Task: Urdu script → Roman Urdu transliteration
• License: CC-BY-NC-4.0, inherited from the base model, plus Cohere Labs' Acceptable Use Policy. Non-commercial use only.

Training Data

Trained on ~80,000 Urdu ↔ Roman Urdu sentence pairs, combining two sources:

• Qasim522/Roman-Urdu-Parl-split (Hugging Face)
• A private curated pairs dataset (Kaggle)

Split as 70,000 train / 5,000 validation / 5,000 test, deduplicated and shuffled before splitting.

Training Procedure

• Method: QLoRA (4-bit NF4 base model, LoRA adapters on attention projections)
• Epochs: 1
• Effective batch size: 16 (batch size 4 × gradient accumulation 4)
• Learning rate: 2e-4
• Warmup steps: 200
• Optimizer: paged_adamw_8bit
• Precision: fp16
• Max sequence length: 224 tokens
• Loss masking: completion-only loss — the instruction and input portion of each example is masked out (label = -100), so the model is only trained to predict the Roman Urdu output, not to reproduce the prompt.

<!-- Optional: add your eval numbers here once available, e.g. ## Evaluation | Metric | Score | |---|---| | BLEU | ... | | BERTScore F1 | ... | Evaluated on the held-out 5,000-example test split. -->

Prompt Format

This model was trained on a strict instruction format. Inference must match this format exactly, or output quality will degrade:

### Instruction:
Transliterate the following Urdu text into Roman Urdu. 
Output ONLY the Roman Urdu. No translation. No explanation.
### Input:
{urdu_text}
### Response:

The model was trained to generate only the Roman Urdu text after ### Response:\n, followed by an end-of-sequence token. It was not trained to add explanations, translations, or any other content.

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "ogx786/urdu-roman-transliterator-tiny-aya-V3"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.eval()

PROMPT_TEMPLATE = """### Instruction:
Transliterate the following Urdu text into Roman Urdu. 
Output ONLY the Roman Urdu. No translation. No explanation.
### Input:
{urdu_text}
### Response:
"""

def transliterate(urdu_text: str, max_new_tokens: int = 128) -> str:
    prompt = PROMPT_TEMPLATE.format(urdu_text=urdu_text)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,          # deterministic output for transliteration
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.eos_token_id,
        )

    # Decode only the newly generated tokens, not the prompt
    new_tokens = output_ids[0][inputs["input_ids"].shape[1]:]
    result = tokenizer.decode(new_tokens, skip_special_tokens=True)

    # Safety net: if the model ever continues past the response
    # (e.g. starts a new "### Instruction:" block), cut it off there.
    result = result.split("### Instruction:")[0].split("### Input:")[0].strip()
    return result


if __name__ == "__main__":
    examples = [
        "کیا حال ہے؟",
        "میں ٹھیک ہوں، شکریہ۔",
    ]
    for urdu in examples:
        print(f"Urdu:  {urdu}")
        print(f"Roman: {transliterate(urdu)}")
        print()

Batch inference

For transliterating many sentences, batch the prompts together rather than calling generate in a loop:

def transliterate_batch(urdu_texts: list[str], max_new_tokens: int = 128) -> list[str]:
    prompts = [PROMPT_TEMPLATE.format(urdu_text=u) for u in urdu_texts]
    tokenizer.padding_side = "left"  # required for correct batched generation
    inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)

    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.eos_token_id,
        )

    results = []
    for i in range(len(urdu_texts)):
        new_tokens = output_ids[i][inputs["input_ids"].shape[1]:]
        text = tokenizer.decode(new_tokens, skip_special_tokens=True)
        text = text.split("### Instruction:")[0].split("### Input:")[0].strip()
        results.append(text)
    return results

Intended Use

• Transliterating Urdu script text (names, addresses, sentences) into Roman Urdu for downstream systems that expect Latin-script input (e.g. voice assistants, search, SMS-based systems).
• Research on low-resource script conversion for South Asian languages.

Limitations

• Trained on 1 epoch over ~70K examples; performance on domains very different from the training data (e.g. poetry, heavy code-switching, informal social media text) is not guaranteed.
• do_sample=False (greedy decoding) is recommended for this task since transliteration should be deterministic; sampling may introduce inconsistency.
• Non-commercial use only, per the base model's CC-BY-NC-4.0 license and Cohere's Acceptable Use Policy.

Citation

If you use this model, please cite the base model:

@misc{tinyaya,
  title = {Cohere Labs Tiny Aya},
  author = {Cohere Labs},
  year = {2026},
  url = {https://huggingface.co/CohereLabs/tiny-aya-global}
}