Gemma3_meme_classification

Finetuned Gemma 3 based Hate Detection in Arabic MultiModal Memes

The rise of social media and online communication platforms has led to the spread of Arabic memes as a key form of digital expression. While these contents can be humorous and informative, they are also increasingly being used to spread offensive language and hate speech. Consequently, there is a growing demand for precise analysis of content in Arabic memes.

This work used Gemma 3 with its vision capability to effectively identify hate content within Arabic memes. The evaluation is conducted using a dataset of Arabic memes proposed in the ArabicNLP ArGuard 2026 challenge. The results underscore the capacity of unsloth/gemma-3-4b-pt fine-tuned with Arabic memes, to deliver the superior performance.

The proposed solutions offer a more nuanced understanding of memes for accurate and efficient Arabic content moderation systems.

Examples of Arabic Memes from ArabicNLP ArGuard 2026 challenge

Examples

Finetuned Gemma 3 Embedding Model with mean pooling

import numpy as np
import torch
import torch._dynamo
from tqdm import tqdm  # Progress bar library
from unsloth import FastVisionModel

from datasets import load_dataset

instruction = "classify meme into Hateful or Not"

def convert_to_conversation(sample):
    
    lis=[]
    lis.append({"type": "text", "text": sample["text"]})
    lis.append({"type": "image", "image": sample["image"]})
                    
    conversation = [
        {
            "role": "system",
            "content": instruction,
        },
        {
            "role": "user",
            "content": lis,
        },
        {"role": "assistant", "content": [{"type": "text", "text": sample["label"]}]},
    ]
    return {"messages": conversation}
pass

dataset = load_dataset("QCRI/ArGuard-Task1",  split="dev_test")

converted_dataset = [convert_to_conversation(sample) for sample in dataset]

# 1. Disable compiler optimization conflicts
#torch._dynamo.config.disable = True
#torch._dynamo.reset()

# 2. Load your fine-tuned model and processor
model_path = "NYUAD-ComNets/Gemma3_meme_classification"
model, processor = FastVisionModel.from_pretrained(
    model_path, 
    load_in_4bit = True,
)
FastVisionModel.for_inference(model)

# Instruction context
instruction = "classify meme into Hateful or Not"

all_embeddings = []
labels_list = []  

num_iterations = len(converted_dataset)

print(f"Starting embedding extraction for {num_iterations} items...")

for idx in tqdm(range(num_iterations), desc="Extracting Embeddings"):
    try:
        sample = converted_dataset[idx]
        
        # Pull text, image, and ground-truth label
        sample_text = sample["messages"][1]["content"][0]["text"]
        sample_image = sample["messages"][1]["content"][1]["image"]
        sample_label = sample["messages"][2]["content"][0]["text"] # From training format

        # Setup multimodal conversation payload
        conversation = [
            {"role": "system", "content": instruction},
            {"role": "user", "content": [{"type": "text", "text": sample_text}, {"type": "image", "image": sample_image}]},
        ]

        # 4. Process inputs normally
        templated_text = processor.apply_chat_template(conversation, tokenize=False)
        inputs = processor(text=templated_text, images=sample_image, return_tensors="pt").to("cuda")


        # 5. Forward Pass
        with torch.no_grad():
            with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
                outputs = model(**inputs, output_hidden_states=True, return_dict=True)

        # 6. Extract final layer and attention mask
        last_hidden_states = outputs.hidden_states[-1]  # [batch_size, seq_len, hidden_dim]
        attention_mask = inputs["attention_mask"]       # [batch_size, seq_len]

        # 7. Masked Mean Pooling (Ignores padding tokens entirely)
        input_mask_expanded = attention_mask.unsqueeze(-1).expand(last_hidden_states.size()).float()
        sum_embeddings = torch.sum(last_hidden_states * input_mask_expanded, dim=1)
        sum_mask = torch.clamp(input_mask_expanded.sum(dim=1), min=1e-9)
        
        all_embedding = (sum_embeddings / sum_mask).squeeze(0).float()
        
        all_embeddings.append(all_embedding.cpu().numpy())


        labels_list.append(sample_label)

    except Exception as e:
        print(f"\nSkipping row {idx} due to an error: {e}")
        continue

embedding_matrix = np.vstack(all_embeddings)

print("Final Concatenated Array Shape:", embedding_matrix.shape)

np.save("test_gemma3_mean_embeddings.npy", embedding_matrix)

Finetuned Gemma 3 for Inference

import numpy as np
import torch
import torch._dynamo
from tqdm import tqdm  
from unsloth import FastVisionModel

# 1. Disable compiler optimization conflicts
torch._dynamo.config.disable = True
torch._dynamo.reset()

# 2. Load your fine-tuned model and processor
model_path = "NYUAD-ComNets/Gemma3_meme_classification"
model, processor = FastVisionModel.from_pretrained(
    model_path, 
    load_in_4bit = True,
)
FastVisionModel.for_inference(model)

dataset = load_dataset("QCRI/ArGuard-Task1", split="dev_test")

ids = []
labels = []

for n in range(500):
    print(n)

    image = dataset[n]["image"]
    text = dataset[n]["text"]

    messages = [
        {"role": "system", "content": instruction},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": text},
                {"type": "image", "image": image},
            ],
        },
    ]

    input_text = processor.apply_chat_template(
        messages,
        add_generation_prompt=True
    )

    inputs = processor(
        images=image,
        text=input_text,
        add_special_tokens=False,
        return_tensors="pt",
    ).to("cuda")

    outputs = model.generate(
        **inputs,
        max_new_tokens=500,
        use_cache=True,
        temperature=0.1,
        top_p=1,
    )

    generated_ids = outputs[:, inputs["input_ids"].shape[1]:]
    result = processor.tokenizer.batch_decode(
        generated_ids,
        skip_special_tokens=True
    )[0].strip()

    print(result)

    ids.append(dataset[n]["id"])
    labels.append(result)
    
    d = pd.DataFrame({
    "id": ids,
    "label": labels,
    "run_id": ["NYUAD_run1"] * len(ids),})

d.to_csv("prediction.tsv", sep="\t", index=False)

We used Low-Rank Adaptation (LoRA) as the Parameter-Efficient Fine-Tuning (PEFT) method for fine-tuning utilizing the unsloth framework.

BibTeX entry and citation info

@misc{aldahoul,
      title={NYUAD at ArGuard Shared Task: Multimodal Embedding Models for
Detecting Arabic Hateful Memes and Unsafe Prompts}, 
      author={Nouar AlDahoul and Yasir Zaki},
      year={2026},
      eprint={},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={}, 
}