Fattah-2.5B-preview

<div align="center">

فتاح — Fattah-2.5B

نموذج لغوي مصري مبني على Qwen3 بتقنية Depth-Up Scaling

Egyptian Arabic LLM Built on Qwen3 with Depth-Up Scaling

</div>

---

Overview

Fattah (فتاح — meaning "the opener" or "the one who opens doors") is a 2.5B parameter Large Language Model specialized for Egyptian Arabic, the most widely spoken Arabic dialect with over 100 million native speakers.

Fattah is built through a novel three-stage pipeline:

1. Depth-Up Scaling (DUS) — expanding Qwen3-1.7B from 28 to 40 transformer layers
2. Continual Pre-Training (CPT) — trained on a ~8.59B token Egyptian Arabic corpus, processing 5.51B tokens (64.1% of the full dataset)
3. Supervised Fine-Tuning (SFT) — 400K Egyptian Arabic instruction-response pairs

⚠️ Note: This is the pre-DPO version (CPT + SFT only). A DPO-aligned version (Fattah-2.5B-v2) is coming soon with improved factual accuracy, reduced hallucination, and better instruction following.

---

Model Details

---

Training Pipeline

Stage 1 — Depth-Up Scaling (DUS)

Starting from Qwen/Qwen3-1.7B-Base, we applied Depth-Up Scaling surgery — the same technique used in SOLAR-10.7B — to expand the model from 28 to 40 transformer layers, increasing parameter count from 1.7B to ~2.5B without any training.

Qwen3-1.7B-Base (28 layers)
        ↓  DUS Surgery
Fattah-DUS (40 layers, ~2.5B)

Layer expansion strategy: concatenate layers [0-23] + layers [4-27], creating a deeper model that inherits the base model's knowledge while providing additional capacity for Egyptian Arabic adaptation.

Stage 2 — Continual Pre-Training (CPT)

Dataset composition:

• 51.7% Egyptian Arabic (web, subtitles, social media, educational)
• 22.1% Modern Standard Arabic (MSA)
• 13.8% English
• 12.4% Code

Stage 3 — Supervised Fine-Tuning (SFT)

Context Extension — YaRN

After SFT, the context window was extended from 32K to 64K tokens using YaRN (Yet another RoPE extensioN):

"rope_scaling": {
    "rope_type": "yarn",
    "factor": 2.0,
    "original_max_position_embeddings": 32768
}

---

Evaluation Results

All evaluations use zero-shot log-likelihood scoring (same methodology as NileChat paper). HellaSwag uses length-normalized accuracy (acc_norm); all other benchmarks use unnormalized accuracy (acc).

Arabic Script Benchmarks — Full Comparison

All evaluations use zero-shot log-likelihood scoring. HellaSwag uses acc_norm (length-normalized accuracy). All other benchmarks use acc (unnormalized accuracy).

Published baselines are from the NileChat paper (Table 1). Fattah rows use our custom evaluation harness with identical zero-shot methodology.

Arabic Script Benchmarks

Key Highlights

• PIQA (61.3%) — Fattah outperforms Qwen2.5-7B (56.4%), gemma-3-4b (56.5%), Llama-3.1-8B (54.3%), and all jais models despite being 2.5B
• WinoGrande (49.4%) — Fattah scores higher than every published baseline in the table, including models 3–5× larger
• Average gap — Fattah post-SFT (40.31%) is behind Nile-Chat-4B (53.93%) by 13.6 points; DPO alignment is expected to close this gap significantly
• Comparable baselines — most fair comparison is with gemma-3-4b-it (4B, 46.74%) — Fattah is 2.5B and pre-DPO, 6.4 points behind a fully aligned 4B model

Full Training Journey (Base → DUS → CPT → SFT)

Key observations:

• DUS surgery caused an expected temporary regression (34.09%) as the new layers were randomly initialized
• CPT recovered and surpassed the base (39.71%), acquiring strong Egyptian Arabic dialect knowledge
• SFT further improved average to 40.31%, with MMLU +1.33% and HellaSwag recovering from 21.4% → 24.0%
• EGY Perplexity improvement of ×2.8 (18.84 → 6.69) confirms deep dialect acquisition during CPT

---

Usage

Installation

pip install transformers>=4.51.0 torch accelerate

Basic Chat

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "belal212/Fattah-2.5B-preview"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "أنت فتاح، مساعد ذكي ومفيد بتتكلم العربي المصري."
    },
    {
        "role": "user",
        "content": "كلمني عن القاهرة"
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False   # disable thinking mode for conversational use
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True
)
print(response)

With Thinking Mode (for complex reasoning)

messages = [
    {
        "role": "system",
        "content": "أنت فتاح، مساعد ذكي بتفكر خطوة بخطوة قبل ما تجاوب."
    },
    {
        "role": "user",
        "content": "ازاي أحسن خوارزمية للـ sorting في Python؟"
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True   # activate <think> mode
)

---

Intended Use

Fattah is designed for:

• ✅ Egyptian Arabic conversational AI
• ✅ Question answering in Egyptian dialect
• ✅ Text generation and creative writing in Egyptian Arabic
• ✅ RAG-based knowledge retrieval systems
• ✅ Foundation for Fattah-Coding (Python + React/TS specialist — coming soon)
• ✅ Agent systems requiring Egyptian Arabic understanding

---

Limitations

• Factual hallucination: As a 2.5B model without DPO alignment, Fattah may confidently generate incorrect facts. A DPO-aligned version is in development.
• Knowledge cutoff: Training data has a knowledge cutoff. Recent events are not known.
• Dialect coverage: Optimized for Egyptian Arabic. Performance on other Arabic dialects is not guaranteed.
• Model size: At 2.5B parameters, Fattah cannot match the factual depth of larger models. Use RAG for knowledge-intensive applications.
• Pre-DPO: This version has not undergone preference optimization. Responses may occasionally be over-cautious or inconsistent in style.

---

Roadmap

---

Training Infrastructure

• GPUs: 2× NVIDIA A6000 48GB
• Framework: ms-swift 4.0.2
• Distributed: DeepSpeed ZeRO Stage 1
• Attention: Flash Attention 2.3.6
• Mixed precision: bfloat16
• Total compute: ~60 GPU-hours (CPT) + ~19 GPU-hours (SFT)

---

Citation

If you use Fattah in your research, please cite:

@misc{fattah2026,
  title        = {Fattah: Egyptian Arabic LLM via Depth-Up Scaling and Continual Pre-Training},
  author       = {Belal},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/belal212/Fattah-2.5B-preview}},
  note         = {Pre-DPO version}
}

---

Acknowledgements

• Qwen Team for the Qwen3-1.7B-Base model
• MBZUAI-Paris for the Egyptian-SFT-Mixture dataset and NileChat benchmarks
• UBC-NLP for the NileChat pre-training corpus
• ms-swift for the training framework

---

<div align="center"> <b>فتاح — بيفتح أبواب الذكاء الاصطناعي للعربي المصري</b><br> <i>Fattah — Opening the doors of AI for Egyptian Arabic speakers</i> </div>