safa-acm-sub-onnx-ensemble

safa-acm-sub-onnx-ensemble

Ensemble classifier for online gender-based violence (OGBV) in Iraqi Arabic social media comments. Four fine-tuned encoders read each comment; their logits are averaged and the argmax over 13 sub-categories is the prediction. The 6 main categories and the binary hate/no-hate view derive from the predicted sub-category through a fixed sub_to_main map. Built by the Jordan Open Source Association (JOSA) with INSM and DRI under the SAFA project.

Architecture

633M parameters total, 1.27 GB. fp16 rather than INT8 because dynamic INT8 quantization collapses BERT-family encoders; fp16 matches the fp32 reference within 0.0001 F1.

Aggregation: equal-weight mean of the four members' raw logits, then argmax. No bias vector. MANIFEST.json is the machine-readable serving contract.

Head: 13-way sequence classification, max_length 128. Each member dir is a self-contained ONNX serving directory (model, tokenizer, training_config.json).

Preprocessing

Inputs must pass the SAFA serving preprocessor before tokenization: strip links, mentions, and photo tags; decode Latin/digit leetspeak; demojize; fold alef variants and alef maqsura; strip diacritics; keep Arabic-script text; cap at 50 words. Each member's training_config.json records its preprocessing flags.

Inference

Per comment: preprocess once, then for each member tokenize with that member's own tokenizer and run its session (MARBERT graphs take token_type_ids), mean the four logit vectors, argmax, and map sub to main. CPU is the intended runtime; the full ensemble serves within a 2 GB budget with no GPU.

Evaluation

Frozen held-out test set, 6,299 comments, never used during development. Scores are on the fp16 ONNX serving path.

Harmful-class macro-F1 (the selection metric): 0.5628, up from 0.530 for the deployed single-model baseline. Escape rate, the share of gold-harmful comments predicted benign: 0.137, down from 0.145.

Training data

The SAFA Iraqi Arabic corpus: 64,040 labeled comments (48,118 general monitoring, 15,922 election period) from ~300 monitored public pages on Facebook, TikTok, and X. Two annotators labeled each comment; a lead annotator adjudicated disagreements. The dataset is gated: thejosango/safa-acm-dataset, access on request.

Intended use and limitations

Built for monitoring, research, and human-review triage. Outputs must feed human decisions; do not use the model for automated enforcement against individuals. Per-class performance drops on rare severe classes (death threats: 14 test examples), and quality degrades outside its domain of public dialectal comments. Confidence scores support triage, and reviewers should stay closest to the rare classes.