arabic-MARBERT-dialect-identification-city-onnx

Arabic MARBERT dialect identification (city-level) - ONNX

ONNX export of Ammar-alhaj-ali/arabic-MARBERT-dialect-identification-city, a MARBERT (UBC-NLP backbone) fine-tune for city-level Arabic dialect identification - finer-grained than the country/region-level dialect identifiers already common on the Hub.

Licence: not declared by the source repository at the time of this conversion. Recorded honestly, not assumed - treat as all-rights-reserved until the author publishes a licence; converted per this org's standing policy that licence ambiguity does not block mirroring.

Provenance note: this is a single-author community fine-tune (264 downloads, 12 likes on the source repo at the time of conversion), not an institutional release. It was spot-checked before conversion effort was invested: it loads correctly and produces plausible, non-degenerate dialect predictions on Gulf/MSA/Egyptian/Levantine/Maghrebi test sentences (see parity section) - it is not a broken or abandoned checkpoint. Treat its accuracy claims with the same caution due any single-author model without institutional validation.

Files

Usage

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer, AutoConfig
from huggingface_hub import snapshot_download

d = snapshot_download("TigreGotico/arabic-MARBERT-dialect-identification-city-onnx")
tok = AutoTokenizer.from_pretrained(d)
config = AutoConfig.from_pretrained(d)
sess = ort.InferenceSession(f"{d}/model.onnx", providers=["CPUExecutionProvider"])

def detect(text):
    enc = tok(text, return_tensors="np", truncation=True)
    inputs = {k: v for k, v in enc.items() if k in [i.name for i in sess.get_inputs()]}
    logits = sess.run(None, inputs)[0]
    idx = int(np.argmax(logits, axis=-1)[0])
    return config.id2label[idx]

print(detect("شلونك اليوم؟ شخبارك؟"))
# -> Doha

For the int8 build, load model.int8.onnx instead.

Parity with the original PyTorch model

5 sentences spanning Gulf, MSA, Egyptian, Levantine and Maghrebi Arabic, compared against the original model's own transformers.pipeline("text-classification", model="...").

parity:
  sample_size: 5
  metric: top1_exact_match
  fp32: 1.00   # 5/5
  int8: 0.80   # 4/5

Differential check: all 5 distinct dialect inputs produced 5 distinct predicted city labels (Doha, MSA, Aswan, Beirut, Tunis) in the fp32 build - no collapse-to-one-label bug.

The one int8 disagreement is a near-miss, not a wrong-dialect-family error: for an Egyptian-dialect greeting the reference predicts Aswan and int8 predicts Cairo - both Egyptian cities, a fine-grained within-country confusion rather than cross-dialect noise.

Caveat on sample size: n=5 is a small smoke-test establishing conversion fidelity, not a systematic accuracy evaluation.