ArGuard-SigLIP-AraBERT-Twitter

ArGuard-SigLIP-AraBERT-Twitter

A cross-attention dual encoder for Arabic hateful meme detection. It pairs the Arabic text encoder AraBERT-Twitter with a frozen SigLIP-L/16 (384 px) image encoder and is trained jointly on both subtasks of ArGuard 2026, Track A:

• A1: binary Hateful / Not Hateful
• A2: multi-label hate categories (10 classes)

It achieves the best A2 test score in our study (macro-F1 0.3851). It was identified after the submission deadline, so it did not contribute to the official ranking. It is described in our paper NAMAA at ArGuard Shared Tasks: Multimodal Fusion, Vision–Language Model Adaptation, and In-Context Prompting for Arabic Hateful Meme Detection (ArabicNLP 2026).

Results

Macro-F1, from Tables 1 and 4 of the paper:

* official shared-task score.

Architecture

meme OCR text ──► AraBERT-Twitter (ft) ──► Linear 768→512 + LayerNorm ──► T ─┐
                                                                               ├─► T' = MHA(T, V, V)   ┐
meme image ─────► SigLIP-L/16 (frozen) ───► Linear 1024→512 + LayerNorm ─► V ─┘   V' = MHA(V, T, T)   │
                                                                                                       ▼
                         h = [masked-mean(T') ; mean(V')]  (1024) ──► LayerNorm ─► Linear 1024→512 ─► z
                         p_A1 = σ(w·z + b)          p_A2 = σ(W z + b)   (10 labels)

• Each bidirectional cross-attention block has 8 heads, d = 512, residual connections, LayerNorm, and a 512→1024→512 feed-forward layer.
• The loss is BCE(A1) + BCE_w(A2), where w holds inverse-frequency positive-class weights.
• The 457.1M parameters are stored in fp32.

Repository contents

Each training_state.pt is a full training state: model (the state dict), optimizer, scheduler, scaler, rng, cfg (the training config), and thresholds / best. For inference, only model, cfg and thresholds are needed.

Decision thresholds (best/)

Thresholds were tuned on the development set by searching 0.05–0.95 in steps of 0.01.

The A2 heads follow the alphabetical label order shown above. Five taxonomy classes (Extremism, Historical, Insults, Stereotyping, Threat) have no training support and are never predicted.

Training

Usage

The checkpoint holds a plain PyTorch state dict. Rebuild the module described above and load the weights from model:

import json, torch
from huggingface_hub import hf_hub_download

REPO = "NAMAA-Space/ArGuard-SigLIP-AraBERT-Twitter"
state = torch.load(hf_hub_download(REPO, "best/training_state.pt"), map_location="cpu", weights_only=False)
cfg, weights = state["cfg"], state["model"]
thr = json.load(open(hf_hub_download(REPO, "best/meta.json")))["thresholds"]

# model = DualEncoder(cfg)            # text_encoder, vision_encoder, proj_t, proj_v, t2i, i2t, cls, head_a1, head_a2
# model.load_state_dict(weights)
# p_a1, p_a2 = model(...)
# hateful = p_a1 >= thr["a1_thr"]
# labels  = [l for l, p, t in zip(LABELS, p_a2, thr["a2_thr"]) if p >= t]

Limitations and ethical considerations

• The model was trained on about 3.5k memes with a single seed. A2 is long-tailed and very sparse: some classes have only a handful of training examples, and per-class thresholds tuned on the development set did not fully transfer to the test set.
• The vision tower is frozen, while the VLM baselines adapt both modalities, so comparisons reflect different training budgets.
• The model handles hateful content and can reproduce social, cultural or dialectal biases. Use it to support culturally informed human moderation, not to replace it. The shared-task data is for research use under its access terms.

NAMAA at ArGuard 2026 models

Citation

@inproceedings{djamai-etal-2026-namaa,
    title     = "{NAMAA} at {ArGuard} Shared Tasks: Multimodal Fusion, Vision{--}Language Model Adaptation, and In-Context Prompting for {A}rabic Hateful Meme Detection",
    author    = "Djamai, Abdelbasset and Al Jallad, Khloud and Emad Eldin, Fatimah and Nacar, Omer",
    booktitle = "Proceedings of the Fourth Arabic Natural Language Processing Conference: Shared Tasks",
    year      = "2026",
    address   = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
}

@inproceedings{alam-etal-2026-arguard,
    title     = "{ArGuard} Shared Task: Harmful Content Detection in {A}rabic Memes and {LLM} Prompts",
    author    = "Alam, Firoj and Biswas, Md. Rafiul and Kmainasi, Mohamed Bayan and Shahroor, Ali Ezzat and Mubarak, Hamdy and Mikros, Georgios and Hasnat, Abul and Zaghouani, Wajdi",
    booktitle = "Proceedings of the Fourth Arabic Natural Language Processing Conference: Shared Tasks",
    year      = "2026",
    address   = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
}

Authors: Abdelbasset Djamai, Khloud Al Jallad, Fatimah Emad Eldin, Omer Nacar (NAMAA Community). Contact: [email protected]