Nawah-50M-RAG-Support-2K

Nawah-50M-RAG-Support-2K

A 51.8M-parameter Arabic (MSA) retrieval-augmented customer-support answerer, trained from scratch. Given a customer question and a handful of knowledge-base passages, it answers from those passages only — and says it does not know when the answer is not there.

It is small enough to run on a CPU (38 MB quantized) and it hallucinates on 4.6% of answers, down from 16.0% after supervised fine-tuning alone.

Quick numbers

Judged by Gemma-4-31B on 500 held-out rows whose companies appear nowhere in training. Score is 0–2 over answerable rows; refusal is measured separately on the 54 unanswerable ones.

For scale: the 31B teacher that generated the training data scores 1.96 and 66.47 chrF++ on the same split. This model reaches 86% of the teacher's judge score at 0.17% of its parameter count, and beats it on chrF++ — the latter because it was trained to mirror the corpus's answer style, not because it is the better model.

Usage

The contract is ChatML: the system turn carries the support preamble plus numbered passages, the user turn carries the customer's question.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "oddadmix/Nawah-50M-RAG-Support-2K"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.float32).eval()

chunks = [
    "يمكن للعميل طلب إغلاق الحساب بتقديم طلب كتابي قبل 15 يوماً من نهاية الشهر الجاري، مع سداد كافة المستحقات المتأخرة.",
    "تمنح الشركة خصماً قدره 15% على الفاتورة السنوية عند تفعيل الدفع التلقائي عبر البطاقة البنكية.",
]
system = (
    "أنت مساعد خدمة عملاء. أجب عن سؤال العميل بالفصحى اعتماداً فقط على المعلومات "
    "التالية. إذا لم تكن الإجابة موجودة في المعلومات، فقل ذلك بأدب واعرض تحويل "
    "العميل إلى أحد موظفي خدمة العملاء."
    "\n\n" + "\n".join(f"[{i}] {c}" for i, c in enumerate(chunks, 1))
)

messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "كيف أغلق حسابي؟"},
]
enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=128, do_sample=False, pad_token_id=1)
print(tok.decode(out[0][enc["input_ids"].shape[-1]:], skip_special_tokens=True))

The system string above is verbatim the one used in training — the preamble, a blank line, then passages numbered [1], [2], … one per line. Grounding degrades if you paraphrase the preamble, drop the numbering, or interleave the question with the passages, so build it exactly this way every time.

GGUF quantizations

Two llama.cpp builds ship in gguf/ for CPU and on-device inference. The ChatML template is embedded, so llama.cpp's /v1/chat/completions works without extra configuration.

llama-server -m nawah-rag-grpo-q8_0.gguf -c 2048

What quantization costs

The judged eval re-run on the same 500 rows with greedy decoding and no repetition penalty in every variant, so the only difference is the weights. These numbers are therefore not comparable to the sampled-decoding table above — compare them only to each other.

Q8_0 is free — identical judge score and refusal accuracy at 45% of the size.

Q4_K_M is probably fine, but this eval cannot prove it is harmless. The F16 conversion is near-lossless (92% of replies byte-identical to bf16) yet still moved the hallucination count by 4 rows, in the opposite direction — so the noise floor on 500 rows is roughly ±4 rows, and Q4's 2-row change sits inside it. The one metric that moved consistently is refusal on unanswerable questions (52/54 → 50/54). Two rows on a 54-row denominator is not significant alone, but refusal is the behaviour GRPO exists to install, and it coincides with Q4 diverging from the reference on a third of all replies. Prefer Q8_0 unless the extra 19 MB matters; if it does, an imatrix-guided Q4 is the next thing to try.

Speed on CPU was the least interesting axis — 16.1 / 17.4 / 19.1 rows/s for F16 / Q8_0 / Q4_K_M. These prompts run ~1,000 tokens with short answers, so the work is dominated by prefill, where quantization helps least. Measured under load from another job; treat it as a lower bound.

How it was built

1. Pretraining. oddadmix/50M-2048-Emhotob — a Llama-architecture Arabic base model trained from scratch at a 2,048-token context.
2. Supervised fine-tuning on arabic-rag-support-25K: 27,427 grounded question/passage/answer rows over 6,873 fictional companies, distilled from Gemma-4-31B. 12% of rows are refusals — a plausible question whose answer is deliberately not in the passages.
3. GRPO round 1 on 1,990 fresh scenarios, company-disjoint from both the SFT train split and the eval split.
4. GRPO round 2 — this checkpoint — continued from the round-1 policy on a 9,959-row pool.

Both GRPO rounds used programmatic rewards with no judge in the loop:

• number grounding — every number in the reply must occur in the gold passages or the question; numbers that appear only in distractor passages, or nowhere at all, are penalized
• refusal correctness — refuse if and only if the answer is absent
• chrF to the gold answer — anchors content and MSA fluency
• sanity — Arabic-script prose of reasonable length

All GRPO training data was generated at scenario indices unused by SFT, and never drawn from the eval set.

The companion multi-turn corpora, arabic-rag-chat-30K and arabic-rag-chat-grpo-5K, are released alongside this model. They did not train this checkpoint — they train the multi-turn members of the same family, and are published so the line is reproducible end to end.

Training ran on a single consumer GPU.

Limitations

• Single-turn only. It has no dialogue training. Give it one question and one set of passages. It will not track a conversation, resolve pronouns across turns, or handle a customer correcting themselves.
• 2,048 tokens is a hard ceiling — roughly 8 passages. There is no graceful degradation past it; positions beyond 2,048 were never trained.
• Modern Standard Arabic only. The corpus is MSA. Dialectal input is out of distribution.
• It answers, it does not compute. The model was trained to quote figures from the passages, never to derive new ones. Do not ask it to add up a bill.
• 4.6% of answers still contain an ungrounded number. This is a 51.8M- parameter model. Do not put it in front of customers without a human path.
• Retrieval quality is the ceiling. Everything above assumes the gold passage is among the ones you supply. This model does not retrieve.
• The judge is one model's opinion. Every score here comes from Gemma-4-31B scoring 500 rows. The hallucination and refusal columns are the more literal measurements; treat the 0–2 score as a comparison between rows of the table, not as an absolute.

Provenance

These weights are the GRPO round-2 policy, published from a pinned, verified revision. model.safetensors has SHA-256 fad8746c8a2ca82907fa09e2ac9c60534ba0ac31a94fa9c72c1832499e2a35fc.

A live demo of this model runs at oddadmix/Nawah-50M-RAG-Support-Demo.

Intended use

Research and prototyping of grounded Arabic question answering: retrieval evaluation harnesses, on-device support assistants, and a baseline for anyone studying how small a grounded RAG answerer can get. It has no general instruction tuning, no safety alignment, and no knowledge of its own — outside the passages you give it, it has nothing to say.

Citation

@misc{nawah50mrag2k,
  title  = {Nawah-50M-RAG-Support-2K: a 51.8M-parameter grounded Arabic RAG answerer},
  author = {Wasfy, Ahmed},
  year   = {2026},
  url    = {https://huggingface.co/oddadmix/Nawah-50M-RAG-Support-2K}
}

© KAND CA 2026 — PROJECT NAWAH