Hannibal-50M-SFT

Hannibal-50M-SFT

Hannibal-50M-SFT is an instruction-tuned (SFT) version of oddadmix/50M-2048-Emhotob, a ~51.8M-parameter Llama-architecture Arabic language model pre-trained from scratch on ~20 billion Arabic tokens with a 2048-token context window.

This model is an experiment to explore whether a language model as small as 50M parameters can learn to follow instructions in Arabic. It is not intended for production use — see Limitations below.

Model Details

• Base model: oddadmix/50M-2048-Emhotob
• Architecture: Llama
• Parameters: ~51.8M
• Context length: 2048 tokens
• Language: Arabic
• Fine-tuning method: Supervised Fine-Tuning (SFT) on chat-formatted instruction data
• License: Apache-2.0

Training Data

The model was fine-tuned on a chat-formatted instruction dataset with the following split:

Each example follows a messages format (list of role/content turns).

Data sources:

1. arbml/CIDAR (~10k rows) — Arabic instruction dataset focused on morphological and grammatical tasks (verb conjugation, noun forms, etc.).
2. ArabicaQA (~59k rows) — Arabic question-answering corpus.

Credits

• Architecture & pretraining recipe: SupraLabs/Supra-50M-Base — "Project Chimera" (Apache-2.0).
• Pre-training corpus: kaust-generative-ai/fineweb-edu-ar.
• Base model: oddadmix/50M-2048-Emhotob.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

BASE_MODEL = "OussamaBenSlama/Hannibal-50M-SFT"

model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    device_map="auto",
    torch_dtype="auto",
)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

messages = [
    {
        "role": "user",
        "content": "ما هي عاصمة تونس؟"
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.1,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Examples

Good responses:

<|user|>
ما هي عاصمة تونس؟</s><|assistant|>
مدينة تونس</s>

<|user|>
أين تقع مدينة قرطاج؟</s><|assistant|>
تقع في منطقة البحر الأبيض المتوسط في شمال أفريقيا</s>

Bad / incorrect responses:

<|user|>
ما هي عاصمة إيطاليا؟</s><|assistant|>
مدينة الفاتيكان</s>

<|user|>
من هو ابن خلدون؟</s><|assistant|>
هو ابن خلدون</s>

These failure cases illustrate the model's current limits: factual errors (wrong capital city) and degenerate/circular answers (repeating the question as the answer) on questions requiring broader world knowledge.

Intended Use

This model is released purely as an experiment to check whether a ~50M-parameter model can learn to follow instructions in Arabic. It demonstrates that small models can pick up basic instruction-following patterns, but is not meant to be used as a reliable assistant or knowledge source. Further training, larger/cleaner datasets, or a larger base model would likely be needed to get meaningfully better results.

Limitations

• Limited Arabic instruction data: High-quality Arabic instruction-tuning datasets are scarce compared to English, limiting the diversity and coverage of tasks the model can learn.
• Noisy training data: Much of the available Arabic instruction data is synthetically generated and not thoroughly cleaned, which is especially problematic for small models — they have little capacity to "average out" noise and are more sensitive to label/data quality issues than larger models.
• Small model capacity: With only ~51.8M parameters, the model has limited world knowledge and reasoning ability, and is prone to factual errors and degenerate outputs (e.g., echoing the question back as the answer).
• Not suitable for factual/knowledge-intensive use cases without further verification of its outputs.

Citation

If you use this model, please also credit the base model and pretraining resources listed in Credits, as well as the underlying datasets:

@inproceedings{alyafeai-etal-2024-cidar,
    title = "{{CIDAR}: Culturally Relevant Instruction Dataset For {A}rabic}",
    author = "Alyafeai, Zaid  and
      Almubarak, Khalid  and
      Ashraf, Ahmed  and
      Alnuhait, Deema  and
      Alshahrani, Saied  and
      Abdulrahman, Gubran  and
      Ahmed, Gamil  and
      Gawah, Qais  and
      Saleh, Zead  and
      Ghaleb, Mustafa  and
      Ali, Yousef  and
      Al-shaibani, Maged",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-acl.764/",
    doi = "10.18653/v1/2024.findings-acl.764",
    pages = "12878--12901",
}
 
@misc{alyafeai2024cidar,
      title={{CIDAR: Culturally Relevant Instruction Dataset For Arabic}}, 
      author={Zaid Alyafeai and Khalid Almubarak and Ahmed Ashraf and Deema Alnuhait and Saied Alshahrani and Gubran A. Q. Abdulrahman and Gamil Ahmed and Qais Gawah and Zead Saleh and Mustafa Ghaleb and Yousef Ali and Maged S. Al-Shaibani},
      year={2024},
      eprint={2402.03177},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
 
@inproceedings{10.1145/3626772.3657889,
author = {Abdallah, Abdelrahman and Kasem, Mahmoud and Abdalla, Mahmoud and Mahmoud, Mohamed and Elkasaby, Mohamed and Elbendary, Yasser and Jatowt, Adam},
title = {ArabicaQA: A Comprehensive Dataset for Arabic Question Answering},
year = {2024},
isbn = {9798400704314},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3626772.3657889},
doi = {10.1145/3626772.3657889},
abstract = {In this paper, we address the significant gap in Arabic natural language processing (NLP) resources by introducing ArabicaQA, the first large-scale dataset for machine reading comprehension and open-domain question answering in Arabic. This comprehensive dataset, consisting of 89,095 answerable and 3,701 unanswerable questions created by crowdworkers to look similar to answerable ones, along with additional labels of open-domain questions marks a crucial advancement in Arabic NLP resources. We also present AraDPR, the first dense passage retrieval model trained on the Arabic Wikipedia corpus, specifically designed to tackle the unique challenges of Arabic text retrieval. Furthermore, our study includes extensive benchmarking of large language models (LLMs) for Arabic question answering, critically evaluating their performance in the Arabic language context. In conclusion, ArabicaQA, AraDPR, and the benchmarking of LLMs in Arabic question answering offer significant advancements in the field of Arabic NLP. The dataset and code are publicly accessible for further research https://github.com/DataScienceUIBK/ArabicaQA.},
booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {2049--2059},
numpages = {11},
keywords = {arabic question answering, information retrieval, llm, question generation},
location = {Washington DC, USA},
series = {SIGIR '24}
}