urdu-bert-64k

Urdu BERT (64k vocab)

A BERT-base model pretrained from scratch on Urdu text, with a custom 64,000-token WordPiece vocabulary. Trained for 20 epochs using the masked language modeling (MLM) objective.

Model Details

• Architecture: BERT-base (12 layers, 768 hidden size, 12 attention heads, 3,072 intermediate size, max position embeddings 512, type vocab size 2)
• Vocabulary size: 64,000 (custom Urdu WordPiece tokenizer, trained from scratch with the HuggingFace BertWordPieceTokenizer, max sequence length 512)
• Pretraining objective: Masked Language Modeling (MLM) only, masking probability 0.15 — no next-sentence prediction (NSP)
• Training epochs: 20
• Language: Urdu (ur)

Training hyperparameters: per-device batch size 24, gradient accumulation steps 16 (effective batch size 384), learning rate 1e-4, weight decay 0.01, warmup steps 10,000, FP16 precision, seed 42. Trained with the HuggingFace Trainer API on a single NVIDIA H100 GPU for approximately 3–4 days.

UrduBERT was validated on two downstream tasks (Named Entity Recognition and Sentiment Analysis), outperforming multilingual BERT (mBERT) on both — detailed results will be released with the accompanying repository.

Intended Uses & Limitations

This is a base pretrained model — it has not been fine-tuned on any downstream task (e.g. classification, NER, question answering). It is intended to be used as a starting point for fine-tuning on Urdu NLP tasks, or for masked-token prediction / contextual embedding extraction as-is.

If you load this model with AutoModel (rather than AutoModelForMaskedLM), you'll see a warning that pooler.dense.weight / pooler.dense.bias are newly initialized. This is expected: the pooler layer was never trained (MLM pretraining does not train it), so it holds random weights until fine-tuned on a downstream task. Do not rely on pooler_output without fine-tuning first — use last_hidden_state instead for embeddings.

How to Use

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

model_id = "farahadeeba/urdu-bert-64k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
results = fill_mask("میں [MASK] کھیل رہا ہوں")

for r in results:
    print(f"{r['token_str']:15s}  score={r['score']:.4f}")

Training Data

The pretraining corpus was compiled from multiple Urdu text sources:

• OSCAR 2019 — the Urdu portion of the OSCAR (Open Super-large Crawled ALMAnaCH coRpus) corpus, filtered from Common Crawl.
• GitHub repositories — publicly available Urdu text datasets.
• Kaggle datasets — publicly available Urdu text datasets.

Sentence-level deduplication was applied across the combined corpus, yielding a final corpus of approximately 5.8 GB of Urdu text. The corpus was split into 5 training files and 1 validation file of approximately 1.16 GB each (~5:1 train/validation ratio).

Citation

This model was developed as part of the following paper :

Farah Adeeba and Miriam Butt. Contextual Embedding Evidence for Main–Light Verb Distinctions in Urdu.

@unpublished{adeeba_butt_urdu_light_verbs,
  title     = {Contextual Embedding Evidence for Main{\textendash}Light Verb Distinctions in Urdu},
  author    = {Adeeba, Farah and Butt, Miriam},
  year      = {2026}
}

License

Apache 2.0