islamqa-retriever-quran

islamqa-retriever-quran

An Arabic bi-encoder for Qur'anic passage retrieval, fine-tuned from NAMAA-Space/AraModernBert-Base-STS through a 5-stage curriculum. This repository holds stage 3 (quqa) of that curriculum — the stage selected as the project's production Qur'an retriever.

Built for the IslamicEval 2025 shared task, Subtask 2 (ArabicNLP @ EMNLP 2025): given a free-text question in Modern Standard Arabic, rank passages from the Thematic Qur'anic Passage Collection (1,266 passages) and Sahih Al-Bukhari (2,254 hadiths).

• Type: sentence-transformers bi-encoder (ModernBertModel + mean pooling)
• Parameters: ~149M · Embedding dim: 768 · Similarity: cosine
• Max sequence length: 512 tokens (tokenizer model_max_length; the backbone's max_position_embeddings is 8192, but 512 is the effective limit — index and query must use the same)
• Pooling: mean · Prompts/prefixes: none (empty for both query and document)
• Language: Arabic (MSA)

---

⚠️ Read this before using it: the Qur'an/Hadith seesaw

This checkpoint is Qur'an-specialised, and it paid for that in hadith recall. The curriculum's per-stage evaluation shows a genuine seesaw between the two corpora — training on Qur'anic data (Tafseer, QuQA) lifts Qur'an recall and depresses hadith recall, and the hadith stage reverses it catastrophically.

Dense-only Recall (no BM25, no reranker), measured on the project's two dev splits:

Qur'an dev = 34 answerable AyaTEC v1.3 dev questions over the 1,266-passage QPC corpus. Hadith dev = 32 questions derived from HaQA over the 2,254-hadith Sahih-Bukhari corpus (a thin, directional split — see caveats below).

Headline: Qur'an dev R@30 0.466 → 0.562 (+20.6% relative) and R@10 0.322 → 0.433 (+34.5%) against the zero-shot base, at the cost of hadith dev R@30 0.750 → 0.531.

So: do not use this model alone on hadith traffic. stage_3 was selected because it dominates every other fine-tuned stage on both splits simultaneously, not because it beats the base model everywhere — it does not.

The recommended production setup is dual: this model + the base model

The parent project runs two encoders, one per corpus, and merges with per-encoder affine calibration constants fitted on shared reference passages:

The calibrated dual beats every single-model configuration on the mixed reading that most closely proxies the hidden test set, and it also passes the hadith dev split. Raw concatenation of the two encoders' scores does not work — the base model's cosine scale sits ~0.27 below this model's, so un-calibrated merging silently suppresses the entire hadith corpus.

End-to-end contribution (full pipeline, Qur'an dev)

Within the complete hybrid pipeline (dense + BM25 → Reciprocal Rank Fusion → NAMAA-Space/GATE-Reranker-V1 cross-encoder → answerability threshold gate):

Over the full 3,520-passage two-corpus store, R@30 rises 0.363 → 0.489. Note that much of the retrieval gain does not reach MAP unless the answerability threshold is re-calibrated for this model's score distribution — a better retriever shifts the distribution the gate sits in.

---

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("IdealisticSolutions/islamqa-retriever-quran")

questions = ["ما هو أعظم الذنوب؟"]
passages  = [
    "إن الله لا يغفر أن يشرك به ويغفر ما دون ذلك لمن يشاء ومن يشرك بالله فقد ضل ضلالا بعيدا.",
    "ألهاكم التكاثر. حتى زرتم المقابر. كلا سوف تعلمون. ثم كلا سوف تعلمون.",
]

q = model.encode(questions, normalize_embeddings=True)
p = model.encode(passages,  normalize_embeddings=True)
print(model.similarity(q, p))   # cosine

No prompt prefix or instruction template is needed — unlike e5 (query: / passage:) or Qwen3, this model was trained without prefixes and adding them will hurt.

Text normalisation must match training. Qur'anic passages (Tanzil simple-clean) carry no diacritics; hadith text does. Strip diacritics ([ً-ْٰ]) from hadith before encoding, and use the identical normalisation at index time and at query time. A retriever that normalises differently on the two sides is a silent quality bug, not an error.

Dual-encoder sketch (Qur'an + hadith)

quran_enc  = SentenceTransformer("IdealisticSolutions/islamqa-retriever-quran")
hadith_enc = SentenceTransformer("NAMAA-Space/AraModernBert-Base-STS")
# Retrieve from each corpus with its own encoder, then merge with per-encoder
# affine calibration (NOT raw score concatenation — see the seesaw section).

---

Training

Curriculum: TyDiQA-ar → Jalalayn Tafseer → QuQA (this checkpoint) → HaQA → task data. Each stage initialises from the previous stage's weights and is evaluated on both dev splits; stage 3 is where the run was cut.

Positive-only training loses roughly 20% recall on this task — the hard negatives are not optional.

Training data sources

• QuQA — Qur'anic question–answer pairs (this stage)
• TyDiQA-ar (stage 1), Jalalayn Tafseer (stage 2), HaQA (stage 4), AyaTEC task data (stage 5)
• Corpora: Thematic Qur'anic Passage Collection v1.1; Sahih Al-Bukhari v1.0

All of these are third-party datasets distributed with the IslamicEval 2025 shared task; their own terms apply to the data, independent of this model's weights.

---

Evaluation caveats (please read)

• The official dev split is Qur'an-only. Every team in the 2025 shared task saw its score collapse from dev to the mixed hidden test set. Never select a checkpoint or a threshold on the Qur'an dev split alone — that trap is exactly why stage_5 (which looks fine on dev) is not the shipped checkpoint.
• The hadith dev split is thin (32 questions) and directional, not precise. It is derived by grounding held-out HaQA questions onto Sahih-Bukhari at a precision-first match threshold; only ~17% ground.
• The hadith dev split contains no unanswerable questions, so it can validate ranking but can never calibrate a zero-answer gate.
• Dense-only numbers above are recall over a single corpus; the end-to-end MAP numbers are over the full two-corpus 3,520-passage store. They are not comparable to each other.
• Sample sizes are small (34 and 32 questions). Treat differences under ~0.02 as noise.

Intended use, and what this model must not be used for

Intended: ranking and retrieving existing, verbatim passages of the Qur'an and of hadith collections in response to Arabic natural-language questions; semantic similarity over Arabic religious text; a retrieval component inside a larger RAG or QA system.

Not intended, and explicitly out of scope:

• This model must never be used to generate, paraphrase, complete, correct, or "modernise" Qur'anic or hadith text. It is an embedding model with no generation head, and the surrounding system should return stored passage IDs and verbatim stored text only. Hallucinated or misattributed religious content is the primary failure mode this entire project exists to prevent.
• It is not a source of religious rulings, and retrieval rank is not a statement of authenticity, authority, or theological correctness.
• It is Arabic-only. Non-Arabic questions must be translated to MSA before encoding; measured end-to-end loss for translated English questions is roughly −13% MAP@10, and about 70% of that loss is translation-induced ranking scatter that no answerability threshold recovers.
• If any translation is displayed to a user, the Arabic source must be shown verbatim and the translation clearly labelled as approximate — never presented as the source text.

Bias and limitations: trained on one Qur'anic QA dataset and one tafseer (Jalalayn), so it inherits their coverage, phrasing, and interpretive perspective. Hadith coverage is limited to Sahih Al-Bukhari. Recall@10 on Qur'an dev is 0.433 — this model is a candidate generator, and the pipeline that produced the MAP numbers above depends on a cross-encoder reranker downstream.

License and attribution

Licensed Apache-2.0, inherited from the base model NAMAA-Space/AraModernBert-Base-STS (Apache-2.0), which is itself a fine-tune of NAMAA-Space/AraModernBert-Base-V1.0. These weights are a derivative work of that checkpoint; credit for the Arabic ModernBERT backbone and its STS fine-tune belongs to NAMAA-Space. The training datasets are third-party and carry their own terms.

Citation

The shared task and the reference pipeline this work reproduces and extends:

@inproceedings{islamiceval2025,
  title     = {IslamicEval 2025 Shared Task, Subtask 2: Qur'an and Hadith Passage Retrieval},
  booktitle = {Proceedings of ArabicNLP @ EMNLP 2025},
  year      = {2025}
}

Backbone: Warner et al., Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder (ModernBERT), 2024. Training framework: Reimers & Gurevych, Sentence-BERT (arXiv:1908.10084).