3arabLM-4B-islamic-v2

<p align="center"> <img src="pic.jpg" width="400"/> <p> <p align="center"> 💜 <a href="https://github.com/sherif1313/3arabLM/"><b>Github</b></a>&nbsp&nbsp | &nbsp&nbsp🤗 <a href="">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp📚 <a href="https://github.com/sherif1313/3arabLM/blob/main/A_Representation_Diagnostic_Framework_for_Arabic_Heritage_Language_Models__A_Case_Study_on_Continued_Pretraining_with_the_Shamela_Corpus%20(2).pdf">Cookbooks</a>&nbsp&nbsp <br> 🖥️ <a href="https://huggingface.co/spaces/sherif1313/3arabLM-4B-v2">Demo</a>&nbsp&nbsp </a> </p>

🕌 3arabLM-4B-Islamic-v2

A Specialized Arabic Language Model for Islamic Heritage

---

📖 Abstract

3arabLM-4B-Islamic-v2 is an ongoing research model dedicated to learning, preserving, recalling, and reconstructing the Arabic Islamic scholarly heritage from classical and authoritative sources. Unlike general-purpose conversational LLMs, the primary objective of 3arabLM is not to imitate everyday conversations or produce short modern summaries. Instead, the project investigates whether a language model can function as a compressed digital library of classical Islamic scholarship, with substantial scholarly knowledge encoded directly into its parameters.

Learn from the books. Preserve the language. Preserve the methodology. Preserve the diversity.

⚠️ **The current model represents less than 5% of the planned continual pretraining schedule. The full project is expected to expand over multiple stages covering nearly the complete Al-Maktaba Al-Shamela ecosystem. .

---

🌟 Project Vision

3arabLM is a long-term research project focused on building a large-scale Arabic language model specialized in the Islamic scholarly heritage. The long-term objective is to gradually expand the model across a broad range of Islamic and Arabic sciences rather than limiting it to a single discipline.

---

📚 Current Domains (Version v2)

The current release has been continued-pretrained on six scholarly domains from the Shamela corpus:

🗺️ Planned Corpus Expansion

Future releases will progressively expand the corpus with additional collections, commentaries, manuscripts, scholarly editions, and specialized literature across Hadith, Tafsir, Fiqh, Arabic linguistics, history, biography, literature, and related fields.

---

🧠 Research Philosophy

The central philosophy of 3arabLM can be summarized as:

Large Language Models as Compressed Digital Libraries

Instead of treating a language model primarily as a text generator, this project investigates whether model parameters can encode substantial amounts of classical scholarly knowledge.

The project therefore explores a different paradigm:

Retrieval from Weights

rather than relying exclusively on:

Retrieval-Augmented Generation (RAG)

The objective is not to eliminate retrieval systems, but to investigate how much scholarly knowledge can be learned and reconstructed directly from the model's internal parameters.

---

📖 Training Philosophy

3arabLM follows a source-oriented training philosophy.

Instead of maximizing general internet-scale diversity, the project focuses on carefully curated scholarly literature.

The model is trained to learn:

• Classical Arabic vocabulary.
• Scholarly terminology.
• Author-specific writing styles.
• Book structures.
• Scholarly argumentation.
• Different methodologies.
• Different schools of jurisprudence.
• Different interpretations.
• Historical context.
• Relationships between scholarly works.

The objective is to preserve the diversity of Islamic scholarship rather than collapsing it into a single generalized representation.

---

🏛️ Preserving Scholarly Diversity

A central objective of this project is to preserve differences between scholars, books, schools, and methodologies.

For example:

• Tafsir al-Tabari ≠ Tafsir Ibn Kathir ≠ Tafsir al-Qurtubi
• Hanafi ≠ Maliki ≠ Shafi'i ≠ Hanbali

The model is intended to learn that different scholars may have:

• Different methodologies.
• Different interpretations.
• Different evidence.
• Different terminology.
• Different jurisprudential positions.
• Different writing styles.

Scholarly disagreement is therefore treated as part of the knowledge itself rather than as noise to be removed.

---

✍️ Classical Arabic Preservation

One of the main objectives of 3arabLM is to preserve the linguistic characteristics of classical Arabic scholarship.

The model is designed to learn:

• Classical Arabic syntax.
• Classical vocabulary.
• Scholarly terminology.
• Traditional expressions.
• Author-specific language.
• Classical argumentation.
• Historical linguistic patterns.

The project investigates whether continual pretraining on large scholarly corpora can produce models capable of reconstructing texts in language close to their original scholarly style.

---

📚 Training Corpus

A major foundation of the project is Al-Maktaba Al-Shamela, together with other Arabic scholarly and heritage sources.

The current and developing corpus covers a broad range of Arabic and Islamic scholarship, including:

📖 Tafsir & Quranic Sciences 📚 Hadith Sciences & Hadith Literature ⚖️ Fiqh & Usul al-Fiqh 🧠 Aqeedah & Islamic Theology 🕋 Sirah & Prophetic Biography 🏛 Islamic History & Civilization 👤 Biography, Tabaqat & Rijal 📝 Arabic Language, Grammar & Morphology 🔤 Lexicography & Dictionaries 📚 Classical Literature & Poetry 🕯 Spiritual & Ethical Literature 📑 Scholarly Research, Bibliographies & Catalogs The corpus is continuously expanding to provide broader coverage of the classical Arabic scholarly tradition and its diverse textual genres. Official Library: goldenshamela

---

🔧 Knowledge Extraction Pipeline

The corpus is transformed through a dedicated extraction and preprocessing pipeline.

                    Shamela
                       │
        ┌──────────────┴──────────────┐
        │                             │
      Books                       Indices
        │                             │
        ▼                             ▼
   Text Parser                  Keyword Parser
        │                             │
        └──────────────┬──────────────┘
                       ▼
            Unified Page Builder
                       │
      ┌────────────┬────────────┬────────────┐
      ▼            ▼            ▼
   Metadata        TOC       Keywords
                       │
                       ▼
                 pages.parquet
                       │
      ┌────────────┬────────────┬────────────┐
      ▼            ▼            ▼
      CPT          SFT          RAG
                       │
        Book → Chapter → Section → Paragraph → Sentence


The preprocessing pipeline aims to preserve the hierarchical structure of scholarly books:


This structure provides contextual information beyond plain text.

---

🧬 Metadata-Aware Training

Future training stages will increasingly incorporate structured metadata.

Planned metadata includes:

The objective is to strengthen the model's understanding of scholarly context and relationships between texts.

---

📊 Preliminary Results

Initial experiments suggest that the model is developing distinguishable internal representations across Islamic scholarly domains.

Key Findings from Representation Diagnostic Analysis

These results indicate that:

• Domain discriminability increases substantially with depth.
• Cross-domain representation similarity decreases sharply in the final layer.
• Stable rank follows a non-monotonic trajectory and reaches a pronounced minimum at the final layer.

This suggests that continued pretraining on Shamela is associated with increasingly specialized final-layer representations, characterized by higher domain discriminability and a more concentrated activation spectrum. ** research paper :**

🧠 Why Memorization?

Modern LLM research frequently relies on Retrieval-Augmented Generation (RAG).

A traditional RAG pipeline can be represented as:
User Question
↓
Retriever
↓
External Knowledge Base
↓
Relevant Documents
↓
Language Model
↓
Answer


3arabLM investigates a complementary direction:
User Question
↓
Language Model
↓
Internal Parameters
↓
Knowledge Recall
↓
Reconstruction

The central research question is:

Can a sufficiently trained language model become a compressed scholarly library?

This project does not claim that external retrieval is unnecessary. Instead, it investigates the capacity of model parameters to store and reconstruct specialized scholarly knowledge.

---

🏗️ Long-Term Architecture

The long-term vision is based on a shared foundation model followed by specialized expert models.

3arabLM Foundation
│
┌─────────┬───────┼───────┬─────────┐
▼      ▼      ▼       ▼       ▼
Fiqh Hadith Tafsir Aqeedah History
Expert Expert Expert Expert Expert
│ │ │ │ │
└─────────┴───────┼───────┴─────────┘
▼
Islamic Heritage
Expert Ecosystem

Potential future experts include:

• Fiqh Expert.
• Hadith Expert.
• Tafsir Expert.
• Aqeedah Expert.
• Islamic History Expert.
• Arabic Language Expert.
• Literature Expert.
• Spiritual Literature Expert.

---

📈 Future Model Scaling

Future research will investigate ways to increase model capacity while preserving previously learned knowledge.

Potential directions include:

• Additional Transformer layers.
• Duplicated upper Transformer blocks.
• Small stochastic initialization.
• Continual pretraining.
• Knowledge-preserving scaling.
• Expert-specialized adapters.
• Metadata-aware training.
• Catastrophic forgetting mitigation.

The goal is to increase memorization capacity while maintaining previously acquired scholarly knowledge.

---

📦 Current Release

Model: sherif1313/3arabLM-4B-islamic-v2

This release represents a new stage in the 3arabLM research project. The previous development focused more narrowly on Islamic scholarly domains such as Fiqh and Tafsir. The new direction expands the training curriculum toward a broader Islamic Heritage Foundation Model covering multiple branches of Islamic and Arabic scholarship. The model should still be considered an early research milestone. A sustantial portion of the planned training curriculum remains to be completed.

These books have been preserved in the model weightsو 

النحو الوافي
تمهيد القواعد بشرح تسهيل الفوائد
شرح ألفية ابن مالك للحازمي
شرح ألفية ابن مالك للشاطبي = المقاصد الشافية
شرح المفصل لابن يعيش
الموسوعة الفقهية الكويتية
موسوعة الإجماع في الفقه الإسلامي
موسوعة فقه العبادات
فتاوى الشبكة الإسلامية
مجموع فتاوى ورسائل العثيمين
السنن الكبرى للبيهقي ت التركي
المحيط في الاحاديث النبوية والسنن والاثار
جامع الرويات
حلية الأولياء وطبقات الأصفياء
صحيح البخاري 
الجامع لشعب الإيمان للبيهقي
الموسوعة العقدية - الدرر السنية
المهذب النقي الجامع لتفسير ابن جرير الطبري
الموسوعة القرآنية
تفسير ابن كثير _ 
تفسير القرطبي
روح البيان

When selecting other books, please modify the code.
            do_sample=True,               
            repetition_penalty=1.08,
            no_repeat_ngram_size=4,

🚀 Usage Example The code must be used because it is compatible with the training method.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# ============================================================
# 🔧 الإعدادات
# ============================================================

MODEL_PATH = "sherif1313/3arabLM-4B-islamic-v2"  # غيّره إلى مسار نموذجك
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
MAX_NEW_TOKENS = 384

# ============================================================
# 📦 تحميل النموذج والمحلل اللغوي
# ============================================================

print("⏳ تحميل النموذج...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)

if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
model.eval()

# ============================================================
# 📝 دالة تنسيق الميتاداتا (مطابقة تماماً للتدريب)
# ============================================================

def format_prefix(book=None, author=None, category=None, part=None, page=None, hierarchy=None):
    """
    توليد الميتاداتا بنفس تنسيق التدريب:
    - جميع الحقول في سطر واحد، مفصولة بمسافات
    - بدون نقطتين (:) بعد اسم الحقل
    - part> و page> و title> (مع >)
    - hierarchy تؤخذ منها القيمة الأخيرة فقط
    """
    tags = []

    if book:
        tags.append(f"book {book}")
    if author:
        tags.append(f"author {author}")
    if category:
        tags.append(f"category {category}")
    if part is not None and str(part).strip():
        tags.append(f"part> {part}")
    if page is not None and str(page).strip():
        tags.append(f"page> {page}")
    if hierarchy:
        parts = hierarchy.split('|')
        title = parts[-1] if parts else ''
        if title:
            tags.append(f"title> {title}")

    return " ".join(tags)

# ============================================================
# 🤖 دالة توليد الإجابة (استرجاع أمين)
# ============================================================

def generate_text(prompt, book=None, author=None, category=None, part=None, page=None, hierarchy=None):
    # بناء الـ prompt بالشكل: "metadata\nالنص"
    prefix = format_prefix(book, author, category, part, page, hierarchy)
    full_prompt = f"{prefix}\n{prompt}" if prefix else prompt

    print(f"\n📝 السؤال/النص: {prompt}")
    print("-" * 50)

    inputs = tokenizer(full_prompt, return_tensors="pt").to(DEVICE)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            do_sample=False,                # استرجاع حتمي (Greedy)
            repetition_penalty=1.02,
            no_repeat_ngram_size=2,
            #repetition_penalty=None
            #no_repeat_ngram_size=None

           
            max_new_tokens=MAX_NEW_TOKENS,
            eos_token_id=tokenizer.eos_token_id,
            pad_token_id=tokenizer.pad_token_id,
        )

    answer = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
    print(f"🤖 إجابة النموذج:\n{answer}")
    print("=" * 80)

# ============================================================
# 📚 أسئلة تستند إلى الكتب الموجودة فعلاً في ./1/2
# ============================================================

TEST_CASES = [

    # ================================================
    # الفقه
    # ========================================================

    {
        "prompt": " حكم المسح على الخفين؟",
        "book": "الموسوعة الفقهية الكويتية",
        "author": "وزارة الأوقاف والشؤون الإسلامية بالكويت",
        "category": "فقه عام"
    },


    
    {
        "prompt": " مبطلات الوضوء؟",
        "book": "موسوعة فقه العبادات",
        "author": "علي بن نايف الشحود",
        "category": "فقه عام"
    },

    
    {
        "prompt": " حكم الجمع والقصر في السفر؟",
        "book": "فتاوى الشبكة الإسلامية",
        "author": "لجنة الفتوى بالشبكة الإسلامية",
        "category": "الفتاوى"
    },


    # ========================================================
    # الحديث والآثار
    # ========================================================

    {
        "prompt": " أول حديث في صحيح البخاري؟",
        "book": "صحيح البخاري - ط الشعب",
        "author": "محمد بن إسماعيل البخاري",
        "category": "متون الحديث"
    },

   
    {
        "prompt": " حديث جبريل في الإسلام والإيمان والإحسان؟",
        "book": "صحيح البخاري - ط الشعب",
        "author": "محمد بن إسماعيل البخاري",
        "category": "متون الحديث"
    },

    {
        "prompt": " فضل طلب العلم؟",
        "book": "الجامع لشعب الإيمان للبيهقي",
        "author": "أحمد بن الحسين البيهقي",
        "category": "متون الحديث"
    },

    {
        "prompt": " فضل الصدق؟",
        "book": "السنن الكبرى للبيهقي ت التركي",
        "author": "أحمد بن الحسين البيهقي",
        "category": "متون الحديث"
    },

    {
        "prompt": " فضل ذكر الله تعالى؟",
        "book": "المحيط في الاحاديث النبوية والسنن والاثار",
        "author": "غير محدد",
        "category": "متون الحديث"
    },

    {
        "prompt": " فضل الصلاة على النبي صلى الله عليه وسلم؟",
        "book": "جامع الرويات",
        "author": "غير محدد",
        "category": "متون الحديث"
    },

    {
        "prompt": " منزلة الصدق في سيرة الصالحين؟",
        "book": "حلية الأولياء وطبقات الأصفياء",
        "author": "أبو نعيم الأصبهاني",
        "category": "متون الحديث"
    },


    # ========================================================
    # العقيدة
    # ========================================================

    
    
    {
        "prompt": " معنى الإيمان بالله؟",
        "book": "الموسوعة العقدية - الدرر السنية",
        "author": "الموسوعة العقدية",
        "category": "العقيدة"
    },


    # ========================================================
    # التفسير
    # ========================================================

    {
        "prompt": " تفسير سورة الفاتحة؟",
        "book": "تفسير القرطبي",
        "author": "محمد بن أحمد القرطبي",
        "category": "التفاسير"
    },

    {
        "prompt": " معنى قوله تعالى: الحمد لله رب العالمين؟",
        "book": "تفسير الرازي = مفاتيح الغيب أو التفسير الكبير",
        "author": "فخر الدين الرازي",
        "category": "التفاسير"
    },

    {
        "prompt": " تفسير قوله تعالى: قل هو الله أحد؟",
        "book": "روح البيان",
        "author": "إسماعيل حقي",
        "category": "التفاسير"
    },
  
    {
        "prompt": " تفسير قوله تعالى: قل هو الله أحد؟",
        "book": "تفسير ابن كثير",
        "author": "ابن كثير",
        "category": "التفاسير"
    },
  
    {
        "prompt": " معنى التقوى في القرآن الكريم؟",
        "book": "الموسوعة القرآنية",
        "author": "غير محدد",
        "category": "التفاسير"
    },

]


# ============================================================
# 🚀 تشغيل الاختبار
# ============================================================

print("\n🚀 بدء اختبار النموذج على أسئلة متنوعة...\n")

for test in TEST_CASES:
    generate_text(
        prompt=test["prompt"],
        book=test.get("book"),
        author=test.get("author"),
        category=test.get("category"),
        part=test.get("part"),
        page=test.get("page"),
        hierarchy=test.get("hierarchy")
    )

print("\n✅ انتهى الاختبار.")

---

⚠️ Current Limitations

The current model has several limitations:

• The training curriculum is still incomplete.
• Some disciplines remain underrepresented.
• Source attribution is still under development.
• Generated claims should be verified against original sources.

---

🔬 Research Directions

The project explores several research questions:

Full-Parameter Memorization

How much classical scholarly knowledge can be encoded directly into model parameters?

Knowledge Reconstruction

Can a language model reconstruct scholarly passages without relying entirely on external retrieval?

Hierarchical Representation

Can لbook, chapter, section, and paragraph structures improve scholarly representations?

Metadata-Aware Pretraining

Can author, book, century, madhhab, and discipline metadata improve knowledge organization?

Knowledge-Preserving Scaling

Can model capacity be increased without catastrophic forgetting?

Expert Models

Can specialized Islamic experts be efficiently developed from a shared foundation?

---

📖 Large Language Models as Compressed Digital Libraries

The central research hypothesis of 3arabLM is that language models may function not only as generators, but also as compressed representations of large scholarly corpora. The project therefore explores the possibility of building:

• 📖 Knowledge Recall Models
• 🧠 Memorization-Oriented Language Models
• 🏛 Compressed Digital Libraries

The ultimate objective is to study how much classical Arabic scholarly knowledge can be represented and reconstructed from model parameters.

---

🎯 Project Goal

The ultimate objective of 3arabLM is:

To preserve, recall, and reconstruct the Arabic Islamic scholarly heritage with the highest possible degree of fidelity while maintaining the language, terminology, methodologies, and diversity of the original scholarly tradition.

The project is not intended to replace scholars.

It is a research effort exploring how modern language-model technology can contribute to:

• Digital preservation.
• Computational study of Islamic heritage.
• Classical Arabic NLP.
• Scholarly knowledge representation.
• Knowledge memorization.
• Text reconstruction.

---

🗓️ Roadmap

---

🤝 Research & Collaboration

The project welcomes researchers and contributors interested in:

• Arabic NLP.
• Islamic heritage digitization.
• Classical Arabic.
• Continual pretraining.
• Knowledge memorization.
• Scholarly text reconstruction.
• Digital humanities.
• Metadata-aware language modeling.
• Islamic manuscript and library digitization.

---

📜 Associated Research

The project is associated with the research work:

A Representation-Diagnostic Framework for Arabic Heritage Language Models: A Case Study on Continued Pretraining with the Shamela Corpus

The research investigates representation diagnostics and domain-specific adaptation of Arabic language models using classical Arabic heritage corpora.

---

🕌 Final Statement

3arabLM explores a simple but ambitious question:

Can a language model become a compressed digital library of the Arabic Islamic scholarly heritage?

The project seeks to preserve not only:

• what scholars wrote,

but also:

• how they wrote,
• how their methodologies differed,
• and how Islamic scholarship developed across centuries.

The long-term vision is a continually expanding Arabic foundation model that represents major branches of Islamic and Arabic scholarly heritage while preserving their linguistic, methodological, and historical diversity.

---

📝 Citation

If you use this model in academic research, please cite the associated research work:

@article{3arabLM-4B-islamic-v2,
    title={3arabLM-4B-islamic-v2},
    author={Hassan, Sherif},
    year={2026},
    url={https://huggingface.co/sherif1313/3arabLM-4B-islamic-v2}
}