Nawah-Router-BERT-6M-bilingual-pretrained

Nawah-Router-BERT-6M-bilingual-pretrained — router head on a genuinely bilingual backbone

Same task as Nawah-Router-BERT-6M-bilingual — give it a text and any categories in English or Arabic, it scores all of them in one forward pass — but the backbone underneath is Nawah-BERT-6M-bilingual, pretrained from scratch on 5B Arabic + 5B English tokens with a tokenizer built for both languages, instead of an Arabic-only-pretrained backbone with a bilingual router head bolted on top.

The comparison this exists to make

Three backbones, same router-head code, same two datasets, scored in one session:

Pretraining the backbone bilingually — not just fine-tuning the head bilingually — closes most of the English gap. Every English split gains 6–11 points over the Arabic-only backbone. Arabic holds flat on three of four splits, with one real cost: unseen_axis (tools/retrieval, held out of training entirely) drops 0.605 → 0.570. That's consistent with the tokenizer trade-off behind the new backbone — Arabic went from 1.20 to 1.37 tokens/word to make room for English — and with the backbone splitting fixed capacity across two languages rather than devoting all of it to one.

Row 1 is still the headline finding underneath all of this: an Arabic-only-pretrained backbone with an English-only router head learns nothing at all (flat loss, chance accuracy). Both fixes here — a bilingual router head, or a bilingual-pretrained backbone — depend on Arabic data being present somewhere in the pipeline.

Usage

from transformers import AutoTokenizer
from routing_model_bilingual import RouterModel, route      # ships in this repo

M = "oddadmix/Nawah-Router-BERT-6M-bilingual-pretrained"
tok = AutoTokenizer.from_pretrained(M)
model = RouterModel.from_pretrained(M)

route(model, tok, "The order is an hour late and the driver isn't answering",
      ["delivery enquiry", "delayed-order complaint", "payment issue"])
# lang="en"/"ar"/"auto" (default) - auto picks by script

Trained on oddadmix/prompt-routing-en (173,223 rows) + oddadmix/arabic-prompt-routing (207,097 rows) mixed in one run, 3 epochs, train_router_head_multi.py defaults (LR 3e-4, batch 32). Each row wrapped in its own language's template before the shared span-pooling head scores it — see that script's docstring.

© KAND CA 2026 — PROJECT NAWAH