AISA-AR-FunctionCall-Think

AISA-AR-FunctionCall-Think

🏷️ Official Track B baseline for the AISA-ArabicFC shared task @ ArabicNLP 2026 (co-located with EMNLP 2026, Budapest)

This model is the organizer-provided baseline for Track B — Reasoning-Augmented Function Calling. It defines the reference score that participating systems are expected to beat. It is released for reproducibility and as a starting point — it is not a competition entry.

A compact (270M-parameter) Arabic function-calling model that, given an Arabic user query (in any of 5 dialects) and a set of candidate tools, writes a short Arabic <think> reasoning trace and then emits a structured tool call. Fine-tuned (LoRA) from google/gemma-3-270m on the AISA-ArabicFC reasoning data.

For the non-reasoning Track A baseline, see the sibling model AISA-AR-FunctionCall-FT.

---

At a glance

---

The shared task

Given an Arabic user query and a set of candidate tool definitions, a system must:

1. Decide whether a function call is required (some queries need no tool),
2. Select the correct function name,
3. Extract the structured arguments,
4. (Track B) Generate an Arabic reasoning trace (<think> … </think>) before the call.

---

How it works — input / output format

This model uses Gemma 3 chat turns with a custom function-calling schema (it does not emit plain JSON). The exact prompt is the text field in the dataset; the structure is:

<bos><start_of_turn>developer
<system instruction in Arabic>
<start_function_declaration>declaration:NAME{description:<escape>…<escape>,parameters:{…}}<end_function_declaration>
…one declaration per candidate tool…<end_of_turn>
<start_of_turn>developer
التاريخ والوقت الحالي …: 2024-04-12T23:05:24
اليوم هو الجمعة
أنت نموذج يمكنه استدعاء الوظائف التالية<end_of_turn>
<start_of_turn>user
أريد مقارنة أسعار تلفاز سامسونج في الأردن<end_of_turn>
<start_of_turn>model

The model then generates:

<think>
يبدو أن نية المستخدم هي الحصول على مقارنة لأسعار تلفاز سامسونج في الأردن. أداة "compare_prices" هي الأنسب …
</think>
<start_function_call>call:compare_prices{country:<escape>Jordan<escape>,product_name:<escape>Samsung TV<escape>}<end_function_call>

For a query that needs no tool, the model omits the <start_function_call> block (→ requires_function = false).

---

Usage

import re, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "TuwaiqAcademy/AISA-AR-FunctionCall-Think"
tok   = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float32, device_map="auto"
).eval()

def parse_model_output(text: str) -> dict:
    """Turn raw generation into the shared-task submission schema."""
    out = {"requires_function": False, "function_name": "none", "arguments": {}, "think": ""}
    if (m := re.search(r"<think>\s*(.*?)\s*</think>", text, re.DOTALL)):
        out["think"] = m.group(1).strip()
    if (m := re.search(r"<start_function_call>\s*call:(\w+)\{(.*?)\}\s*<end_function_call>", text, re.DOTALL)):
        out["requires_function"] = True
        out["function_name"] = m.group(1)
        for key, str_val, num_val in re.findall(r"(\w+):(?:<escape>(.*?)<escape>|([^,}]+))", m.group(2)):
            val = str_val if str_val else num_val
            try:
                val = float(val) if "." in str(val) else int(val)
            except (ValueError, TypeError):
                pass
            out["arguments"][key] = val
    return out

# Easiest path: take the ready-made prompt from the dataset's `text` field and
# cut it at the model turn (everything after is what the model should produce).
from datasets import load_dataset
row = load_dataset("TuwaiqAcademy/AISA-ArabicFC", split="validation")[0]
prompt = row["text"].split("<start_of_turn>model\n")[0] + "<start_of_turn>model\n"

inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.no_grad():
    gen = model.generate(**inputs, max_new_tokens=250, do_sample=False)  # greedy
raw = tok.decode(gen[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)

print(parse_model_output(raw))
# → {'requires_function': True, 'function_name': 'compare_prices',
#    'arguments': {'country': 'Jordan', 'product_name': 'Samsung TV'},
#    'think': 'يبدو أن نية المستخدم …'}

The parsed dict maps directly onto a leaderboard submission line: {"id", "tool_called", "arguments", "think"} (use function_name → tool_called).

---

Evaluation

Scored on the AISA-ArabicFC held-out test set (1,000 positive + negative examples) using the official v2 metrics:

• FnAcc — function-name accuracy over all samples (also penalises hallucinated / missed calls; negatives have gold none)
• ArgEM — strict argument exact match, over positives only
• ThinkRate — fraction of outputs with a non-empty <think> trace
• Overall (Track A) = 0.40·FnAcc + 0.60·ArgEM
• Overall (Track B) = 0.30·FnAcc + 0.50·ArgEM + 0.20·ThinkRate

Baseline results

• Think-Before-Call rate (ThinkRate): 0.868 for this model; 0.000 for all non-reasoning baselines.
• Hallucination rate: 0.000 on negative (no-tool) queries.

Key takeaways

• 🎯 Argument extraction is the open challenge. Tool selection is largely solved (FnAcc ≈ 0.98), but strict argument exact match tops out at 0.541 — and GPT-4o reaches only 0.070 zero-shot. This is where the task is won or lost.
• 🪶 A 270M model beats GPT-4o across every metric here, showing the value of task-specific Arabic training and lowering the compute barrier to entry.
• 🗣️ Cross-dialect gaps remain. FnAcc varies by roughly 10–15 points across dialects, with Gulf and Levantine consistently the hardest and Maghrebi (small sample) the easiest — see the Track C diagnostic in the task overview paper.

---

Training

• Base: google/gemma-3-270m
• Method: LoRA (rank 64), 3 epochs, cosine LR scheduler
• Data: AISA-ArabicFC training split (~10.5K examples) with 12,000 Arabic reasoning annotations for the <think> traces
• Objective: produce a short Arabic reasoning trace followed by a single structured tool call (or no call for negatives)

---

Intended use & limitations

Intended use

• A reference baseline to compare against and reproduce for the AISA-ArabicFC shared task.
• A lightweight starting point for Arabic tool-use / agentic experiments.

Out of scope / limitations

• Trained for the 27-tool, 8-domain AISA-ArabicFC schema and its prompt format; behaviour on arbitrary tools or free-form chat is undefined.
• Single-turn, single-call setting — no multi-tool or multi-turn dialogue.
• Argument extraction is imperfect (ArgEM 0.541): expect errors in date normalisation, numeric typing, and dialectal argument phrasing.
• Uneven dialect coverage (Maghrebi is only ~1.3% of data); robustness varies by dialect.
• A 270M model — capacity-limited by design to keep the baseline accessible.

---

Related resources

• 🏆 Shared task page: https://huggingface.co/spaces/Omartificial-Intelligence-Space/AISA-ArabicFC-Shared-Task
• 📊 Leaderboard: https://huggingface.co/spaces/TuwaiqAcademy/AISA-ArabicFC-SharedTask-Leaderboard
• 📚 Dataset (train + dev): TuwaiqAcademy/AISA-ArabicFC

---

Citation

@inproceedings{najar2026aisaarabicfc,
  title     = {AISA-ArabicFC: Arabic Function Calling for Agentic AI Systems},
  author    = {Najar, Omar},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  year      = {2026}
}

License

This model is a derivative of Gemma 3 and is distributed under the Gemma Terms of Use. By using it you agree to those terms and to the Gemma Prohibited Use Policy. The AISA-ArabicFC dataset is released separately under Apache-2.0.

Contact

Shared-task organizers — [email protected] · Tuwaiq Academy