alexandriax-arat5v2-balanced

AlexandriaX-2026 · Subtask 1 — AraT5v2-base, dialect-rebalanced full fine-tune

English to dialectal Arabic dialogue translation over 13 Arabic varieties, from the NAMAA Community work on AlexandriaX-2026 (ArabicNLP 2026 / EMNLP). Same architecture and recipe as alexandriax-arat5v2-base, but fine-tuned on a temperature-rebalanced country distribution, so that the under-represented dialects (TN, MA, EG, YE, MR) get a larger share of the gradient without adding any new data.

[!CAUTION] The weights in this repo are void — do not use them for inference. The training run was silently poisoned by a tokenizer fallback: the model was fine-tuned for 10 epochs against targets in which every Arabic character had been replaced by <unk>. This is not a packaging problem and no tokenizer swap at load time recovers it. The recipe itself is sound, so this card documents it in full and ships train_arat5v2_balanced.py with the bug fixed and three hard gates added, ready to retrain. For a working AraT5v2 checkpoint, use alexandriax-arat5v2-base (25.12 dev / 23.26 blind spBLEU).

Post-mortem: how the run was voided

The notebook asked for AraT5v2's slow SentencePiece tokenizer:

tokenizer = T5Tokenizer.from_pretrained("UBC-NLP/AraT5v2-base-1024",
                                        legacy=True, use_fast=False)

On that session's stack this raised argument 'vocab': 'dict' object cannot be converted to 'Sequence'. It is not an exotic failure — the same error reproduces on any current transformers 5.x install, because the unified T5Tokenizer tries to rebuild a Unigram model from spiece.model and fails. Control fell through to the notebook's except branch:

tokenizer = T5Tokenizer.from_pretrained("t5-base", legacy=True)   # 32,100 English tokens

t5-base's vocabulary has no Arabic coverage. It encodes صباح الخير as [3, 2, 3, 2, 3, 2, ...] — space, <unk>, space, <unk> — which decodes back to nothing but spaces. Every training target was destroyed before the first optimiser step, so ten epochs taught the model to predict <unk> and spaces, and that is exactly what it emits:

{"turn_order": 1, "prediction": ".          ."}
{"turn_order": 2, "prediction": "."}

The dev scoring run on those predictions returned 0.0 spBLEU / 1.86 chrF++. A second consequence: embedding rows above index 32,100 — most of the 110,208-token vocabulary — were never touched by a gradient.

Why this matters beyond one checkpoint. A tokenizer fallback is a silent failure: training loss still decreases smoothly (predicting <unk> is easy), no exception is raised, and the damage only surfaces at scoring time. The fixed script therefore refuses to proceed unless three conditions hold: the tokenizer's vocabulary exceeds 100k tokens, an Arabic probe string round-trips without <unk>, and fewer than 10% of the first tokenized target's tokens are <unk>. A post-training generation smoke test asserts the output contains Arabic characters.

The tokenizer in this repo has since been replaced

The repo now ships AraT5v2's real vocabulary — tokenizer.json (110,100 tokens, verified token-id-identical to spiece.model + </s>), spiece.model, and a tokenizer_config.json that loads on transformers 4.x and 5.x alike. The wrong t5-base files are gone, so anything you retrain from this repo starts from a correct tokenizer.

This does not repair the weights. Loading them now simply produces different garbage instead of the training-time garbage, because what was destroyed was the training targets:

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("NAMAA-Space/alexandriax-arat5v2-balanced")
model = AutoModelForSeq2SeqLM.from_pretrained(
    "NAMAA-Space/alexandriax-arat5v2-balanced", torch_dtype=torch.bfloat16)
model.tie_weights(); model.eval()
# ...expect punctuation and blanks. Documented, not a bug in your setup.

Intended use

The recipe and the post-mortem, not the weights. Retrain with the bundled script if you want to test whether flattening the country distribution helps a small encoder-decoder on low-resource Arabic varieties — that question is still open.

---

The shared task

AlexandriaX-2026 (ArabicNLP 2026 / EMNLP) — Context-Aware Dialectal Arabic MT and MT Evaluation. This model was built for Subtask 1: Context-Aware English-to-Dialectal Arabic Dialogue Translation.

Given one English dialogue turn together with its conversation history and metadata — target country/dialect, domain, participant roles, speaker, and speaker→addressee gender direction — the system must produce the turn in the requested country's spoken Arabic, preserving meaning while adapting lexical, morphological, pragmatic and sociolinguistic choices to that variety.

Two tracks: constrained (provided data only, ≤5B parameters) and unconstrained (any external data or model). Ranking is by spBLEU (primary) and chrF++ (secondary), each macro-averaged over countries.

Official data (UBC-NLP/alexandria)

Split sizes in turns, as published by the organisers:

Libyan (LY) and Sudanese (SD) appear only at test time — they are zero-shot for every system trained on this data.

Conversation-level counts: 21,146 train / 3,963 dev / 4,706 public-test conversations; mean 3.13 turns per conversation (range 1–5). Mean length 102 characters of English source, 74 characters of dialectal target.

Dialects (13 countries). Egyptian, Jordanian, Lebanese, Libyan, Moroccan, Mauritanian, Omani, Palestinian, Saudi, Sudanese, Syrian, Tunisian, Yemeni. Labels are country + sub-dialect, and several countries carry more than one: Palestinian 10 (Nabulsi and Albira urban, plus Falahi varieties of Surif, Kobar, Noba, Ni'lin, Shuqba, Aboud, Silwad, Ramallah), Omani 5 (Suri, Rustaqi, Al-Wafi, Ibri, Seebi), Saudi 3 (Southern, Hijazi, Khaleeji), Yemeni 3 (Taiz, San'ani, Central), Syrian 2 (Levantine Standard, Homsi). The remaining countries carry one label each (e.g. Egyptian Arabic (Cairene), Moroccan Standard Darija, Mauritanian Hassaniya, Libyan Arabic (Misrati/Central)).

Domains (11, near-uniform). Agriculture and farming, Commerce and transactions, Construction and real estate, Education and academia, Energy and resources, Everyday and social, Healthcare and medical, Legal and financial, Logistics and transportation, Professional and workplace, Tourism and hospitality.

Speaker direction (turns, train+dev+public test): female→male 30,636 · male→female 30,203 · male→male 20,465 · female→female 11,868. The corpus carries 76 distinct translator IDs and 44 reviewer IDs.

Code-switching in the gold is strongly dialect-specific — the share of gold turns containing Latin characters runs from TN 39.1% / MA 33.8% / LB 18.1% down to SY 1.2% / YE 0.8%. Systems that normalise every borrowing into Arabic script are penalised hardest on Maghrebi references (see Known limitations).

Evaluation protocol

• spBLEU — sacrebleu.BLEU(tokenize="flores200"), corpus-level per country, then averaged over countries.
• chrF++ — sacrebleu.CHRF(word_order=2), same averaging.
• Decoding is turn-by-turn: at turn n the conversation history contains the system's own previous outputs, never the gold ones. (An early evaluation harness in this project leaked gold previous-turn Arabic into the prompt and inflated scores by ≈2.4 spBLEU; every number reported here comes from the corrected, self-conditioned harness.)

---

Training data — the rebalancing recipe

Same 63,130 official training turn pairs as the base sibling, resampled rather than extended. Country probabilities are flattened by temperature sampling and then clipped:

TEMP = 3.0        # p_c ∝ (n_c / N) ** (1 / TEMP)
FLOOR = 4000      # minimum turns per country
CAP = 12000       # maximum turns per country
SEED = 42

The total stays at 63,130 turns, so training cost is unchanged; only the mixture moves. No back-translated or monolingual data is used here.

Net effect: Palestinian drops from 22.5% to 12.7% of the mixture while Tunisian rises from 3.1% to 6.6% — a 2.1x upsample of the scarcest variety and a 1.8x downsample of the most abundant one. LY and SD remain absent (test-only) and are handled zero-shot.

Training procedure — every hyperparameter

Extracted verbatim from AlexandriaX_NB11_AraT5v2_Balanced.ipynb. Runnable single-file version, with the tokenizer bug fixed: train_arat5v2_balanced.py.

Model and data

Optimisation

Precision and hardware

Bookkeeping

Inference

Results

No valid evaluation exists, and none can be produced from these weights.

For context, the checkpoint it should be compared against — alexandriax-arat5v2-base, same architecture, unbalanced data — scores 25.12 dev spBLEU / 40.66 chrF++ and 23.26 / 39.03 on the private blind test, with its weakest countries being exactly the ones this rebalancing targets (MR 15.21, MA 19.75, YE 20.84 on dev).

Known limitations

• The weights are void, for the reason given in the post-mortem. Retraining is the only fix.
• Unvalidated hypothesis. Nothing here shows that temperature rebalancing helps or hurts; the question is untested, not answered.
• Resampling adds no information. Upsampling repeats existing Tunisian and Moroccan turns; it cannot supply the vocabulary those varieties are missing, and it raises the risk of overfitting the repeated turns across 10 epochs.
• Inherits every limitation of the base sibling: no conversational context, one prefix per country (sub-dialects unaddressed), aggressive normalisation of Latin-script borrowings where Maghrebi gold keeps them, LY/SD zero-shot, and metric-only evaluation.

---

Where this model sits in the NAMAA system

All Subtask-1 systems built by the team, scored on the official 12,250-turn dev set (11 countries) and, where they were run, on the 14,459-turn private blind test (13 countries). Country-macro spBLEU / chrF++.

¹ That run was trained against destroyed targets — a tokenizer fallback substituted t5-base (32,100 English tokens) for AraT5v2's 110,208-token vocabulary, so every Arabic character became <unk>. It scored 0.00 spBLEU and cannot be recovered without retraining; the post-mortem and a fixed training script are in its card.

² That run stopped at step 2,500 of a planned 31,568 (epoch 0.63 of 8) and was never decoded on the development set, so no score exists for it. Its card carries the full recovered configuration.

Two findings from this bank of models are worth carrying elsewhere.

1. Parameter count does not predict rank below the cap. The 368M encoder–decoder AraT5v2 beats every larger decoder-only fine-tune on identical data, and among the decoder-only models spBLEU falls Qwen2.5-1.5B > NileChat-3B > Gemma-3-1B — the reverse of their size order. A reading consistent with this: the metric rewards fidelity to the annotators' conventions over generative fluency. A translator fine-tuned on the provided targets acquires those conventions; a decoder-only model several times its size contributes fluency n-gram overlap does not credit.
2. Combination is not free. Fitted and evaluated on disjoint halves of the dev conversations: routing per country +0.07, per country + sub-dialect +0.28, per country + domain −0.32, MBR consensus over 5 systems −0.57, MBR over the top-2 per dialect −0.81 — against a best single system of 24.98. The per-turn oracle reaches 32.25 (+7.27), so the right output is usually in the pool and the failure is in selection: three NileChat variants agree with one another and outvote the single strongest system, so consensus weights model-family size rather than quality. The submitted system therefore routes per dialect under a ±0.40 spBLEU margin guard instead of voting.

---

The collection

All released artefacts live in NAMAA at AlexandriaX-2026:

Every model repo above carries a single-file train_*.py reproduction script with the exact hyperparameters that produced its checkpoint; the dataset repo carries build_backtranslated_pairs.py.

Official task data: UBC-NLP/alexandria. Base models: UBC-NLP/AraT5v2-base-1024, UBC-NLP/NileChat-3B-Base, facebook/nllb-200-1.3B, google/mt5-large.

---

Team

NAMAA Community — Fatimah Emad Eldin (Cairo University) · Omer Nacar (Tuwaiq Academy) · Khloud Al Jallad (Arab International University) · Mona Abdelazim (Ain Shams University).

Citation

Coming soon. The NAMAA system-description paper for AlexandriaX-2026 is under review for the ArabicNLP 2026 (EMNLP) proceedings; this card will be updated with the final ACL Anthology reference and DOI when the proceedings are published. Until then, please cite as:

@inproceedings{namaa-alexandriax-2026,
  title     = {{NAMAA} Community at {AlexandriaX-2026}: Prompting, Fine-Tuning and Agreement
               Voting for Dialectal Arabic Translation and Evaluation},
  author    = {Emad Eldin, Fatimah and Nacar, Omer and Al Jallad, Khloud and Abdelazim, Mona},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  note      = {To appear. Citation coming soon.}
}

Please also cite the shared task and the base model:

@inproceedings{alexandriax2026,
  title     = {{AlexandriaX-2026} Shared Task: Context-Aware Dialectal Arabic Machine
               Translation and MT Evaluation},
  author    = {El Mekki, Abdellah and Elmadany, AbdelRahim A. and Magdy, Samar M. and
               Ezzini, Saad and El-Haj, Mo and Jarrar, Mustafa and El-Beltagy, Samhaa and
               Abbas, Mourad and Zaraket, Fadi and Al Mandhari, Salim and Alyafeai, Zaid and
               Ghanem, Bernard and Abdul-Mageed, Muhammad},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  note      = {Overview paper. Citation coming soon.}
}

Acknowledgements

Thanks to the AlexandriaX-2026 organisers for the data, the evaluation infrastructure and their responsiveness during the evaluation phases.