zabaanai-pakistan-multilingual

🇵🇰 ZabaanAI - Pakistan Multilingual Language Model

An open-source, CPU-friendly AI model for Pakistan's major languages

🌐 Supported Languages

🏗️ Base Model: google/mt5-small

We chose mT5-small over XLM-R and mBERT because it's the only architecture that natively supports all required tasks:

*Encoder-decoder achieves 47% lower first-token latency and 4.7x higher throughput on CPU vs decoder-only (Microsoft 2025).

🎯 Supported Tasks

All tasks use mT5's unified text-to-text format:

📂 Repository Structure

├── scripts/
│   ├── 01_collect_datasets.py        # Download 35+ HF datasets
│   ├── 02_clean_text.py              # Clean, deduplicate, normalize
│   ├── 03_train_tokenizer.py         # Extend mT5 tokenizer
│   ├── 04_prepare_training_data.py  # T5 span corruption prep
│   ├── 05_train_cpu.py               # CPU-optimized LoRA pretraining
│   ├── 06_finetune_lora.py           # LoRA fine-tuning (5 tasks)
│   ├── 07_evaluate.py                # BLEU, ROUGE, perplexity
│   ├── 08_quantize.py                # ONNX INT8 / 8-bit quantization
│   └── 09_deploy_space.py           # HF Space deployment
├── docs/
│   ├── 01_DATASET_COLLECTION.md      # Full dataset sourcing guide
│   ├── 02_TOKENIZER_GUIDE.md         # Tokenizer improvement strategy
│   ├── 03_TRAINING_GUIDE.md          # CPU training configuration
│   ├── 04_FINETUNING_GUIDE.md        # LoRA fine-tuning recipes
│   └── 05_DEPLOYMENT_GUIDE.md        # 5 deployment options
├── deployment/
│   ├── app.py                        # Gradio web app (6 tasks)
│   ├── api_server.py                 # FastAPI REST API
│   ├── Dockerfile                    # Docker container
│   └── requirements.txt              # All dependencies
├── notebooks/
│   └── 01_explore_datasets.ipynb     # Dataset visualization
├── tokenizer/                        # Extended tokenizer (250,141 vocab)
├── config.yaml                       # Central configuration
├── setup.py                          # Environment setup
├── QUICKSTART.md                     # 10-step quick start
└── README.md                         # This file

⚡ Quick Start

1. Install Dependencies

pip install transformers datasets tokenizers accelerate peft sentencepiece
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install protobuf tiktoken gradio optimum[onnxruntime]

2. Collect Datasets

python scripts/01_collect_datasets.py

3. Clean & Merge

python scripts/02_clean_text.py

4. Extend Tokenizer

python scripts/03_train_tokenizer.py

5. Prepare Training Data

python scripts/04_prepare_training_data.py

6. Start CPU Training

python scripts/05_train_cpu.py --max_steps 50000 --batch_size 2 --lora_r 64

7. Fine-Tune for Tasks (GPU recommended)

python scripts/06_finetune_lora.py --task translation --batch_size 8
python scripts/06_finetune_lora.py --task sentiment --batch_size 8

8. Deploy

python scripts/09_deploy_space.py --username your-username --space_name zabaanai-demo

🔧 CPU Training Configuration

Training time: ~70 hours for 50K steps on 8-core CPU, or ~3 hours on T4 GPU.

🎯 Task Fine-Tuning

Translation

python scripts/06_finetune_lora.py --task translation
# Datasets: Humair332/Vast-Urdu (10M pairs), Kashmiri-English, Pashto-English

Sentiment Analysis

python scripts/06_finetune_lora.py --task sentiment
# Datasets: community-datasets/roman_urdu, urdu_sentiment_corpus

Summarization

python scripts/06_finetune_lora.py --task summarization
# Datasets: News articles with headlines as pseudo-summaries

Question Answering

python scripts/06_finetune_lora.py --task qa
# Datasets: Sindhi encyclopedia, generated Q&A pairs

Chatbot / Conversational AI

python scripts/06_finetune_lora.py --task chatbot
# Datasets: Urdu-Instruct, Sindhi SFT, Pashto Alpaca

🚀 Deployment Options

📊 Data Mixture for Pretraining

Urdu:           35%  (news, poetry, social media, Wikipedia)
Sindhi:         20%  (large corpus available)
Pashto:         15%  (ZamAI corpora)
Punjabi:        10%  (news, transliteration)
Roman Urdu:      8%  (social media, tweets)
English:         7%  (prevent catastrophic forgetting)
Kashmiri:        3%  (smaller corpus)
Balochi:       1.5%  (very limited data)
Saraiki/Hindko/Brahui: 0.5%  (augmented from related)

🔤 Tokenizer

mT5's native tokenizer already covers all Pakistan scripts with 250,000 tokens + byte fallback. We extended it with 52 Pakistan-specific tokens:

• Cities: لاہور, کراچی, اسلام آباد, پشاور, کوئٹہ
• Names: محمد, علی, فاطمہ
• Institutions: سپریم کورٹ, نیشنل اسمبلی
• Roman Urdu: bhai, yaar, theek

Vocabulary size: 250,141 (was 250,100)

📈 Evaluation

python scripts/07_evaluate.py --model_path training/checkpoints/final --task translation
# Metrics: BLEU (translation), ROUGE (summarization), Perplexity (generation)

💾 Quantization for Production

# ONNX INT8 (recommended for CPU)
python scripts/08_quantize.py --model_path training/checkpoints/final --method onnx

# 8-bit (for GPU)
python scripts/08_quantize.py --model_path training/checkpoints/final --method 8bit

Benefits: 60-70% smaller model, 2-3x faster inference on CPU.

🤝 Contributing

We welcome contributions! Priority areas:

• More datasets for Saraiki, Hindko, Brahui
• Better evaluation benchmarks
• Fine-tuned task-specific adapters
• Documentation translations

📜 License

Apache 2.0 (same as mT5 base model)

🙏 Acknowledgments

• Google Research for mT5
• ZamAI for Pashto datasets
• aakashMeghwar01 for Sindhi corpus
• Humair332 for Vast-Urdu parallel corpus
• Omarrran for Kashmiri datasets
• community-datasets for Roman Urdu sentiment data

📚 Citation

@software{zabaanai2025,
  title = {ZabaanAI: Pakistan Multilingual Language Model},
  author = {Open Source Contributors},
  year = {2025},
  url = {https://huggingface.co/shaikhsalman/zabaanai-pakistan-multilingual}
}

---

Built with ❤️ for Pakistan's languages 🇵🇰