iqbal-urdu-transformer

Urdu Poetry Transformer (Trained from scratch)

Small GPT-style transformer trained from scratch on Urdu ghazals from the Rekhta dataset.

Model Details

• Architecture: Custom TinyGPT (causal self-attention transformer)
• d_model: 128
• num_layers: 3
• num_heads: 4
• block_size: 128
• vocab_size: 2767
• Parameters: 1,302,272

Training

• Dataset: Rekhta Urdu ghazals (9709 poems, 231,092 tokens)
• Best validation loss: 5.2026
• Perplexity: 181.74

Limitations

Trained on a small corpus (~4MB text); generation quality reflects this scale and should not be expected to produce fully coherent poetry.

Usage

This model uses a custom architecture and is NOT compatible with AutoModel.from_pretrained(). You need the TinyGPT class definition to load it:

from safetensors.torch import load_file
import json

with open("config.json") as f:
    config_dict = json.load(f)

config = TinyGPTConfig(**config_dict)
model = TinyGPT(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)