Nawah-RuleCheck-5M

Nawah-RuleCheck-5M — مدقّق قواعد عربي بـ 20.3 ميجابايت

A 5,080,704-parameter Arabic rule checker; weights file 20,328,000 bytes (20.33 MB). Give it a text and a rule written in ordinary Arabic and it answers مطابق (the text satisfies the rule) or مخالف (it does not).

بالعربية: نموذج عربي صغير (5,080,704 معامل) يأخذ نصًا وقاعدة مكتوبة بلغة طبيعية ويقرّر إن كان النص مطابقًا للقاعدة أم مخالفًا لها.

5 transformer layers, hidden size 128. It is a sequence classifier (LlamaForSequenceClassification), not a generator — the answer is two-valued, so a classification head fits better at this size and is directly scorable.

Where the parameters are. 984,704 of the 5,080,704 parameters are the transformer body; 4,096,000 (80.6%) are the 32,000 × 128 embedding table. Small Arabic models are mostly vocabulary — worth knowing before reading the results as reasoning capacity.

The size ladder

This model is one rung of a four-point ladder trained on identical data and scored by one harness in one session (eval_all.py). Nothing below is carried over from an earlier run or another card.

CPU ms = single example, batch 1, float32, 2 threads, median of 200 runs, same input for every model. On an RTX 5090 this model is 1.80 ms.

What the four columns mean

• seen wording (6,624 pairs) — held-out texts (task_id split, zero text overlap with training), rules phrased the way training phrased them. Text generalisation.
• unseen wording (6,624 pairs) — the same held-out texts, rule phrasings held out of training entirely, drawn from the same 631-paraphrase pool.
• minimal pairs (1,080 pairs) — a real text and a surgically edited copy whose verdict flips, verified by rules_common.py. Any cue that merely correlates with the label inside the corpus dies here.
• hand-written wording (1,200 pairs) — 10 rules restated in terse, colloquial Arabic written from scratch, sharing almost no vocabulary with the paraphrase pool. This is the honest hard column, and it is where model size actually buys something.

Per-rule accuracy, unseen wording

Per-rule accuracy, hand-written wording

Limitations

Read the hand-written wording column, not the headline. Inside the paraphrase distribution the whole ladder is bunched between 0.9778 and 0.9980 — a 2.0-point spread across a 100x size range. On rule wordings written from scratch the same four models spread from 0.5558 to 0.8883, a 33.2-point gap, and the 500K rung falls below the 0.5925 majority-class baseline. Almost all of what size buys on this task is robustness to wording you did not train on. If your rules are a fixed catalogue you can phrase in the training register, the small rungs are close to free. If users will phrase rules in wording you do not control, size matters and no rung on this ladder is finished.

Trained on 19 rules decidable from the string itself. Rules needing world knowledge, judgement or multi-step inference are out of distribution. Texts are 1–3 line Arabic business documents (classified ads, support tickets, job posts, complaints, rental listings) across nine regions; longer or very different text is untested. Word-count rules are the weakest family — counting is the one operation here that cannot be pattern-matched.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

M = "oddadmix/Nawah-RuleCheck-5M"
tok = AutoTokenizer.from_pretrained(M)
model = AutoModelForSequenceClassification.from_pretrained(M).eval()

def check(text, rule):
    x = tok(f"النص: {text}\nالقاعدة: {rule}", return_tensors="pt", add_special_tokens=False)
    with torch.no_grad():
        p = model(**x).logits.softmax(-1)[0]
    return model.config.id2label[int(p.argmax())], float(p.max())

check("للبيع سيارة نظيفة، للتواصل على الرقم 0551234567",
      "يُمنع ظهور أي رقم هاتف (8-15 خانة) في النص")

Input format is النص: {text}\nالقاعدة: {rule} with add_special_tokens=False. Text first, rule second: the head pools the last non-pad token, so under causal attention only the trailing rule tokens can attend to the whole text.

Training

Base oddadmix/Emhotob-5M-v2 — Llama, 5 layers, hidden 128, vocab 32,000, tied embeddings, 2048 context. 159,240 (text, rule) pairs, LR 1e-4, 10 epochs, cosine schedule, 500 warmup steps, effective batch 64, bf16, max_length 224. Best checkpoint selected on unseen-wording macro-F1.

Learning rate across the ladder

"Smaller models need a higher learning rate" is true here — but only below 5M. Each rung was swept independently on the same data and the same eval:

The optimum climbs 60× between the 5M and the 500K (1e-4 → 6e-3), and getting it wrong is expensive: on the 500K base the big model's recipe scores 0.9179 unseen against 0.9780 tuned — six points that look like a capacity limit and are not.

Above 5M the effect simply stops. The 5M and the 51.8M want the same 1e-4, and pushing the 5M up toward the 500K's learning rate destroys it (3e-3 → 0.88, 6e-3 → 0.77 unseen). So neither recipe transfers in either direction: the tiny rungs need their own LR, and that LR must not be carried back up the ladder.

This rung's sweep

Same data, same eval splits, same everything — only LR and epoch budget move:

The expected direction reverses here. At 500K the best LR is 6e-3; at 5M it is 1e-4 - lower than the 51.8M model's 3e-4, and 60x lower than the 500K's. Above 1e-3 this base degrades fast (3e-3 -> 0.88, 6e-3 -> 0.77 unseen), so copying the tiny model's recipe up the ladder would have been as wrong as copying the big model's recipe down it. 1e-4 is bracketed on both sides (3e-5 -> 0.9893, 3e-4 -> 0.9846).

How the labels were made

Labels are computed, not model-judged. Each rule is a deterministic predicate over the raw string (rules_common.py, shipped here), so ground truth is exact by construction. That also makes paraphrasing free supervision: rewording a rule cannot change its verdict.

Siblings

• oddadmix/Nawah-RuleCheck-500K — 518,256 params
• oddadmix/Nawah-RuleCheck-1M — 1,073,504 params
• oddadmix/Nawah-RuleCheck-5M — 5,080,704 params
• oddadmix/Nawah-RuleCheck-v2 — 51,787,264 params

Reproducing every number on this page

python eval_all.py oddadmix/Nawah-RuleCheck-500K oddadmix/Nawah-RuleCheck-1M \
                   oddadmix/Nawah-RuleCheck-5M  oddadmix/Nawah-RuleCheck-v2

eval_all.py is in this repo, along with the frozen case sets it scores against (cf_cases.json, ood_cases.json) so the minimal pairs and the hand-written wordings are byte-identical for every rung and every re-run. Also here: prepare_rules_cls.py, train_rules_cls.py, gen_rule_paraphrases.py, counterfactual_eval.py, rules_common.py (which defines the labels), and the 631 rule paraphrases in rule_paraphrases.json.