Downloads · 30 days
0
Urdu-Bert
mahwizzzz/Urdu-Bert
Urdu-Bert is a machine learning model from mahwizzzz. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
I implemented BERT model from scratch with Pytorch on Urdu Data. It uses Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks, just like the original BERT model.
Access
Public
Updated Aug 7, 2025
Repo size
11.1 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt11.1 MB · 99%
At a glance
- License
- mit
- Access
- Public
- Created
- Aug 7, 2025
- Updated
- Aug 7, 2025
- SHA
- c618a409
Model card
From the Hugging Face model README
Urdu-BERT Pretraining (PyTorch)
I implemented BERT model from scratch with Pytorch on Urdu Data. It uses Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks, just like the original BERT model.
✅ Features
- Trained on Urdu-1M-news-text
- Custom WordPiece tokenizer
- Multi-head attention & transformer encoder blocks
- NSP and MLM heads
- Uses PyTorch and HuggingFace Tokenizers
- Training tracked with Weights & Biases (WandB)
⚙️ Training Setup
| Setting | Value |
|---|---|
| Epochs | 20 |
| Batch Size | 64 |
| Sequence Length | 64 tokens |
| Embedding Size | 128 |
| Encoder Layers | 2 |
| Attention Heads | 2 |
| Max LR | 2.5e-5 |
| Warmup Steps | 1000 |
| Optimizer | Adam |
📊 Training Loss (WandB)
Below is an example of the loss curve during training:
This shows MLM loss, NSP loss, and total loss reducing over time.
📌 Notes
- NSP uses random sentence pairs for false samples
- Positional embeddings are created using sin/cos
- Custom learning rate scheduler with warm-up
- Model is built completely from scratch (no pretrained weights)
📚 References
- License
- mit
- Languages
- ur
- Created
- Aug 7, 2025
- Updated
- Aug 7, 2025