supra-50m-arabic-sft-mix

Supra-50M-Arabic-SFT (mixed)

Instruction-tuned (SFT) version of a from-scratch 50M-parameter Arabic language model. The base is a Llama-style decoder (hidden 512 / 12 layers / 8 heads, 4 KV / ctx 1024) pretrained on Arabic Wikipedia + FineWeb-2 Arabic with a custom 32k Arabic BPE tokenizer. This checkpoint adds a ChatML chat format and instruction-following behaviour.

⚠️ Arabic-only. The model has never seen English (or other languages) in pretraining or SFT. Do not expect translation to/from English or non-Arabic output.

Training data (balanced mixture)

Datasets are mixed and shuffled (not trained sequentially, to avoid catastrophic forgetting). All of each dataset is used; the small native high-quality set (CIDAR) is upsampled so it isn't drowned out. The Cap column shows any row limit (- = full).

Split: 1,527,470 train / 15,429 eval.

Chat format

ChatML, with loss on assistant turns only (assistant_only_loss):

<|im_start|>user
ما هي عاصمة مصر؟<|im_end|>
<|im_start|>assistant
القاهرة.<|im_end|>

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Abdelkareem/supra-50m-arabic-sft-mix")
model = AutoModelForCausalLM.from_pretrained("Abdelkareem/supra-50m-arabic-sft-mix")
messages = [{"role": "user", "content": "اشرح ما هو الذكاء الاصطناعي بكلمات بسيطة."}]
enc = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.2)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Training configuration

• Objective: causal LM, assistant-only loss (prompt masked)
• Epochs: 3.0 · effective batch 32 · max length 1024
• LR: 2e-05 cosine, warmup ratio 0.03, weight decay 0.0
• Precision: bf16 · Trainer: TRL SFTTrainer

Intended use & limitations

Research / educational demo of a tiny Arabic instruction model. At 50M parameters it learns chat format and simple tasks (short answers, summaries, rewrites) but:

• hallucinates facts and cannot do reliable multi-step reasoning,
• is Arabic-only (no English / translation),
• is not suitable for production or any high-stakes use.

Dataset licenses are those of the source datasets; review them before redistribution.