LFM2.5-8B-A1B-fastagent-toolcaller

LFM2.5-8B-A1B — Fast-Agent Tool-Caller (rebalanced)

A QLoRA fine-tune of LFM2.5-8B-A1B (uncensored/abliterated base) specialized for sharp, contract-clean function calling with correct abstention. Ships as a LoRA adapter and a Q4_K_M GGUF.

TL;DR — measured on a self-hosted BFCL-style AST eval (official bfcl-eval checker, non-live categories, dotted-name policy):

⚠️ These are NOT official BFCL leaderboard numbers. They cover only the 5 non-live AST categories (simple / multiple / parallel / parallel_multiple / irrelevance), scored with the official bfcl-eval AST checker under the dotted function-name policy (gorilla-openfunctions-v2 config, underscore_to_dot=False) which matches this model's dotted interface. They are internally comparable (same harness across all rows), not leaderboard-equivalent. See "Honest evaluation notes" below.

What it is / intended use

A tool-calling agent model. It reliably emits Hermes/Qwen-style <tool_call>{...}</tool_call>, selects the right tool, does clean parallel calls, and abstains on creative/irrelevant requests. Built to drive agentic pipelines (single + parallel function calls), not as a general chat assistant.

Training

• Method: QLoRA (4-bit NF4), r=16, target_modules=all-linear, ~600 steps, seq 2048.
• Response-only loss masking (mask system/user/tool turns; train only the assistant turn + <|im_end|>) — this is the fix that stops over-generation, hallucinated <tool_response>, and <think> loops.
• Data (~21.3k examples): Salesforce/xlam-function-calling-60k (8k gold positives), MadeAgents/xlam-irrelevance-7.5k (4k — the rebalance), NousResearch/hermes-function-calling-v1 (func-calling-singleturn + glaive-5k), interstellarninja/hermes_reasoning_tool_use, databricks/databricks-dolly-15k (prose-answer-with-tools, over-call guard).
• The two-round story: adding 8k pure-positive xLAM examples lifted tool-calling (+10–17 pts on simple/multiple/parallel) but crashed abstention (irrelevance 95→63.5). Doubling the irrelevance counterweight (2k→4k) recovered abstention (→80–82.5) while keeping the tool-calling gains — the shipped model is that rebalance.

Behavior (live, on the Q4 GGUF via llama.cpp, CPU)

• ✅ Clean single <tool_call>; correct tool selection; clean parallel calls (multiple <tool_call> blocks).
• ✅ Abstains on creative requests (answers a poem in prose, doesn't force a tool).
• ✅ strict == permissive in eval → honors the <tool_call> wrapper contract.
• ⚠️ On a plain factual question with tools present, it tends to reply "I don't have a tool that can do that" rather than answering from knowledge — an artifact of the abstention training. Fine for a tool-agent role, less ideal as a general assistant.
• ⚡ ~2–3 s/call on CPU once resident (LFM2.5-8B-A1B is an MoE, ~1B active).

Usage

GGUF (llama.cpp / ollama):

llama-server -m lfm-rebal-Q4_K_M.gguf -c 8192 --jinja --port 8080
# then POST OpenAI-style chat with a <tools>...</tools> system prompt; it returns <tool_call>{...}</tool_call>

LoRA adapter (peft): apply adapter_model.safetensors on top of blockblockblock/LFM2.5-8B-A1B-uncensored-abliterated.

Honest evaluation notes (the audit trail)

This model was evaluated adversarially; two scoring bugs were found and corrected, which is why the numbers are trustworthy:

1. A first custom checker was too lenient (accepted extra args) → inflated a number to 75.4% — withdrawn.
2. The official checker was then mis-invoked with an unregistered model_name → KeyError on 43% of dotted-name cases scored them wrong → deflated to 46.9% — withdrawn. Fixed by using a registered dotted-policy config. The 77.0 / 82.8 / 84.5 figures are the corrected, dotted-policy, official-bfcl-eval-AST results. An adapter-vs-base guardrail confirmed the skill is adapter-learned (base ~0–1% on tool categories, enabled 77–83%), not a load artifact.

Limitations

• Uncensored/abliterated base — no safety alignment; use responsibly.
• Over-abstains on factual-general questions (see Behavior).
• Numbers are non-live AST categories only, not full BFCL; no live/multi-turn/executable evaluation.

Attribution & license

• Base: blockblockblock/LFM2.5-8B-A1B-uncensored-abliterated (abliteration of LiquidAI/LFM2.5-8B-A1B). Governed by the LFM Open License v1.0 — attribution required; review its commercial terms before commercial use.
• Training datasets credited above (Salesforce xLAM, MadeAgents, NousResearch Hermes, interstellarninja, Databricks Dolly).