sawb-marbertv2

Sawb — MARBERTv2 / CAMeLBERT-MSA-half

Part of the Sawb Arabic Cultural Hallucination Detection Collection for ICAIRE 2026 Track 3.

Overview

Sawb — MARBERTv2 / CAMeLBERT-MSA-half is one of four Arabic encoder models in the Sawb 4-Model Ensemble — the primary competition submission achieving binary macro F1 = 0.9647 on the 457-example validation set. Each model is a binary classifier for detecting cultural hallucinations in Arabic LLM outputs.

Multi-dialect Arabic pre-training — particularly valuable for detecting dialectal confusion hallucinations.

A cultural hallucination occurs when an LLM produces a response that is factually or culturally incorrect within Arab/Islamic contexts — misapplying Western legal frameworks (EU AI Act, GDPR) to Islamic jurisprudence, fabricating hadith or Islamic rulings, ignoring Arab institutional contributions to AI (KACST, SDAIA, MBZUAI, Vision 2030), responding in the wrong Arabic dialect, or using Western examples in Saudi/Gulf contexts.

Ensemble (Primary Submission)

The full Sawb detect-then-explain pipeline:

1. 4-Model Ensemble Detection: Average hallucination probability from all four models (this model + sawb-arabert + sawb-arabert-large + sawb-arbertv2 + sawb-marbertv2) at threshold θ = 0.30 → Binary Macro F1 = 0.9647
2. Explanation Generation: DeepSeek API (exp16 dialectal few-shot prompt, 30 parallel threads) generates case-specific Arabic explanations for each detected hallucination

Model Architecture

Training

Evaluation Results

Usage — Solo Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("HassanB4/sawb-marbertv2")
model = AutoModelForSequenceClassification.from_pretrained("HassanB4/sawb-marbertv2")
model.eval()

question = "كيف تُطبَّق مبادئ أخلاقيات الذكاء الاصطناعي في القضاء الإسلامي؟"
answer = "يجب تطبيق AI Act الأوروبي على المحاكم الإسلامية..."

text = f"السؤال: {question}\n\nإجابة النموذج: {answer[:500]}"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    logits = model(**inputs).logits

prob = torch.softmax(logits, dim=-1)[0, 1].item()
is_hallucination = prob > 0.30
print(f"Hallucination probability: {prob:.3f} | Detected: {is_hallucination}")

Usage — 4-Model Ensemble (Primary Submission)

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

MODELS = [
    "HassanB4/sawb-arabert",
    "HassanB4/sawb-arabert-large",
    "HassanB4/sawb-arbertv2",
    "HassanB4/sawb-marbertv2",
]
THRESHOLD = 0.30

question = "كيف تُطبَّق مبادئ أخلاقيات الذكاء الاصطناعي في القضاء الإسلامي؟"
answer = "يجب تطبيق AI Act الأوروبي على المحاكم الإسلامية..."
text = f"السؤال: {question}\n\nإجابة النموذج: {answer[:500]}"

probs = []
for model_name in MODELS:
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForSequenceClassification.from_pretrained(model_name)
    model.eval()
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
    with torch.no_grad():
        logits = model(**inputs).logits
    probs.append(torch.softmax(logits, dim=-1)[0, 1].item())

ensemble_prob = sum(probs) / len(probs)
is_hallucination = ensemble_prob > THRESHOLD
print(f"Ensemble probability: {ensemble_prob:.3f} | Detected: {is_hallucination}")

Hallucination Categories

Dataset

HassanB4/sawb-arabic-hallucination-dataset — 1,828 Arabic (question, LLM answer) pairs covering 6 cultural hallucination categories.

Collection

Sawb Arabic Cultural Hallucination Detection Collection