urdu-finance-embeddings

<div align="center">

🏦 FinGuard Urdu Finance Embeddings

The first domain-specific embedding model for Pakistani financial queries across Urdu, Roman Urdu, and English

</div>

---

The Problem This Solves

Pakistan has 220 million people, millions of active fintech users (Easypaisa, JazzCash, Raast), and a growing Islamic finance sector — yet almost no NLP infrastructure exists for Pakistani financial queries.

The challenge is threefold:

• Pakistanis write finance questions in three mixed forms: Urdu script (کیا زکوٰۃ واجب ہے), Roman Urdu (zakat ka hisab kaise karein), and English (how to calculate zakat)
• Islamic finance has specialized vocabulary (Murabaha, Riba, Musharakah, Nisab) that generic multilingual models misunderstand
• Existing models trained on Wikipedia/news fail completely on Pakistani financial context

This model was built from the ground up to fix that.

---

What Makes This Different

---

Model Performance

Evaluated using InformationRetrievalEvaluator on a held-out validation set of 147 financial queries across all 8 categories.

Baseline (zero fine-tuning): Accuracy@1 ≈ 72.8% → +21 points gained from domain training

---

Quick Start

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("hassan7272/urdu-finance-embeddings")

# Works in all three forms — no preprocessing needed
queries = [
    "zakat ka hisab kaise karein",                    # Roman Urdu
    "زکوٰۃ کا حساب کیسے کریں",                        # Urdu script
    "how to calculate zakat in Pakistan",             # English
]

# Your answer corpus
answers = [
    "Zakat 2.5% hoti hai jo nisab se zyada savings par lagti hai...",
    "Meezan Bank Islamic saving account mein profit milta hai...",
]

query_embeddings  = model.encode(queries,  convert_to_tensor=True)
answer_embeddings = model.encode(answers,  convert_to_tensor=True)

scores = util.cos_sim(query_embeddings, answer_embeddings)
print(scores)

RAG Retrieval (FAISS)

from sentence_transformers import SentenceTransformer
import faiss, numpy as np

model = SentenceTransformer("hassan7272/urdu-finance-embeddings")

# Build index over your answer corpus
answers = ["answer 1 ...", "answer 2 ...", ...]
embeddings = model.encode(answers, normalize_embeddings=True)

index = faiss.IndexFlatIP(embeddings.shape[1])   # inner product = cosine on normalized
index.add(embeddings.astype(np.float32))

# Retrieve
query = "Easypaisa se ghar ka kiraya kaise bharein"
q_emb = model.encode([query], normalize_embeddings=True).astype(np.float32)
scores, indices = index.search(q_emb, k=10)      # top-10 retrieval

---

Training Pipeline

This model was produced through a full ML engineering pipeline built from scratch — not just a fine-tuning script.

Stage 1 — Dataset Engineering

The training data comes from hassan7272/urdu-finance-qa, a custom 1,510-record Q&A dataset across 8 Pakistani financial categories.

Text normalization handled:

• Urdu Unicode character variants (multiple encodings of the same letter)
• Roman Urdu repeated character collapse (kyaaaa → kyaa)
• Spacing/punctuation normalization across all three languages
• Language detection per record (ur / roman_ur / en)

Stage 2 — Hard Negative Mining

Instead of random negatives, a category-aware hard negative miner was built:

Query: "zakat ka hisab kaise karein"  (Islamic Finance)
Random negative: "online bill payment kaise karein"   ← too easy, model ignores
Hard negative:   "loan ka interest kaise calculate hota hai"  ← looks relevant, is WRONG

The confusable category map forces the model to discriminate on intent, not just topic. For example, islamic_finance negatives are drawn from loans_credit and personal_finance — the most semantically similar but semantically incorrect categories. Keyword overlap filtering ensures negatives are hard but not trivially different.

Stage 3 — Training Strategy

2,752 training examples were built from three sources:

• Primary: (question_ur, answer_ur) pairs — native Urdu / Roman Urdu
• Hard negatives: (question_ur, hard_negative, answer_ur) triplets — explicit confusion signal
• Cross-lingual (30%): (question_en, answer_en) pairs — free cross-lingual alignment

Loss: MultipleNegativesRankingLoss (scale=20, cosine similarity)

• Every other pair in the batch becomes an automatic in-batch negative
• With 16 batch size → each example sees 15 automatic negatives + 1 explicit hard negative
• Model learns to push correct pairs together and wrong pairs apart in embedding space

Training config:

Base model    : paraphrase-multilingual-mpnet-base-v2
Epochs        : 4
Batch size    : 16
Learning rate : 5e-5
FP16          : True (mixed precision)
Platform      : Kaggle GPU (CUDA)
Train time    : 7 minutes

Stage 4 — Evaluation

InformationRetrievalEvaluator was used at every 86 steps during training, treating the entire validation answer set as a retrieval corpus. This is the same evaluation setup used in real RAG systems — not a toy cosine similarity test.

Training progression:

Model converged at epoch 2 with train loss of 0.063 — a very low value indicating clean convergence without overfitting on the 1.5k dataset.

---

Dataset Coverage

---

Architecture Context

This model is Phase 1 of the FinGuard RAG system — a full retrieval-augmented generation pipeline for Pakistani financial advisory:

User Query (Urdu / Roman Urdu / English)
        ↓
  Query Normalization
        ↓
  Hybrid Search (this model + BM25)
        ↓
  Top-10 Retrieval
        ↓
  MMR (diversity filtering)
        ↓
  Cross-Encoder Reranker → Top-3
        ↓
  LLM Answer Generation

The embedding model (this model) handles the vector search component with 93.9% accuracy at Top-1.

---

Limitations

• Trained on synthetic Q&A data — real-world distribution may differ slightly
• Coverage is Pakistan-specific; Indian Urdu financial context may vary
• Answers are from 2024 — regulatory/rate information may be outdated
• Roman Urdu spelling variation is partially handled but highly informal text may still vary

---

Citation

If you use this model in your research or application, please cite:

@misc{hassan2025finguard,
  title        = {FinGuard Urdu Finance Embeddings: Domain-Specific Multilingual Embeddings for Pakistani Financial RAG},
  author       = {Hassan},
  year         = {2025},
  publisher    = {HuggingFace},
  url          = {https://huggingface.co/hassan7272/urdu-finance-embeddings},
  note         = {Fine-tuned on urdu-finance-qa dataset with hard negative mining and MultipleNegativesRankingLoss}
}

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title     = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author    = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    year      = "2019",
    url       = "https://arxiv.org/abs/1908.10084",
}

---

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.3
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.8.3

---

<div align="center"> Built as part of the <strong>FinGuard RAG</strong> project — Islamic Finance Advisory for Pakistan 🇵🇰 </div>