sawb-multilingual-e5

Sawb — Multilingual-E5-Large

Part of the Sawb Arabic Cultural Hallucination Detection Collection for ICAIRE 2026 Track 3.

Overview

Sawb — Multilingual-E5-Large is a binary classifier for detecting cultural hallucinations in Arabic LLM outputs. Fine-tuned from intfloat/multilingual-e5-large (560M parameters) — a cross-lingual embedding model pre-trained on multilingual text pairs. This model serves as a non-Arabic-specific baseline with cross-lingual generalization capabilities.

A cultural hallucination occurs when an LLM produces a factually or culturally incorrect response within Arab/Islamic contexts — misapplying Western legal frameworks, fabricating religious references, or responding in the wrong Arabic dialect.

Model Architecture

Training

Evaluation Results

Despite its cross-lingual (non-Arabic-specific) pre-training, mE5-Large achieves strong performance (F1=0.9467). This is comparable to the Arabic-specific ARBERTv2 (0.9457) and MARBERTv2 (0.9264), but below AraBERT base (0.9599) and AraBERT-Large (0.9788 at optimal threshold).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("HassanB4/sawb-multilingual-e5")
model = AutoModelForSequenceClassification.from_pretrained("HassanB4/sawb-multilingual-e5")
model.eval()

question = "كيف تُطبَّق مبادئ أخلاقيات الذكاء الاصطناعي في القضاء الإسلامي؟"
answer = "يجب تطبيق AI Act الأوروبي على المحاكم الإسلامية..."

text = f"السؤال: {question}\n\nإجابة النموذج: {answer[:500]}"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    logits = model(**inputs).logits

prob_hallucination = torch.softmax(logits, dim=-1)[0, 1].item()
is_hallucination = prob_hallucination > 0.30

print(f"Hallucination probability: {prob_hallucination:.3f}")
print(f"Is hallucination: {is_hallucination}")

Dataset

Trained on HassanB4/sawb-arabic-hallucination-dataset.

Collection

Sawb Arabic Cultural Hallucination Detection