toolflow-qwen3-8b-utility-v2

ToolFlow Qwen3-8B Utility — v2 (LoRA adapter)

A QLoRA adapter on top of Qwen/Qwen3-8B, fine-tuned for tool-calling against a 20-tool utility catalog. Within 10 pp of Claude Sonnet 4.6 on the same eval at ~16× lower estimated per-task serving cost.

This is the production adapter from the ToolFlow project — a five-week sprint training a small open model to do enterprise-style structured tool calling competitively with frontier APIs.

Headline results

Evaluated on a 270-case eval set (BFCL v3 prompts re-labeled against the utility catalog by Qwen3-235B, anti-circularity guard against same-vendor evaluation):

Statistical significance vs Sonnet (paired bootstrap, 10000 resamples, McNemar): Δ = -10.0 pp, 95% CI [-16.7, -3.3], McNemar p = 0.048. The gap is real but small in absolute terms, and dominated by one specific failure mode — see "Known limitations" below.

What this adapter does

Given a user request and a system prompt enumerating available tools, emits a JSON tool call (or {"tool": "ABSTAIN", "arguments": {}} if no tool fits):

User: What's the area of a circle with radius 4?
Model: {"tool": "math.area_circle", "arguments": {"radius": 4}}

User: Schedule a meeting with my team tomorrow at 3pm.
Model: {"tool": "ABSTAIN", "arguments": {}}

The 20-tool catalog covers math, stats, geo, finance, web, text, auth, science utilities. Full catalog at tools.json in the repo.

Training details

Training data

4,461 examples, mixed:

• 3,749 positive tool-call examples filtered from Salesforce/xlam-function-calling-60k (CC-BY-4.0) and glaiveai/glaive-function-calling-v2 (Apache 2.0), with field-rename + schema-repair to map upstream tool conventions onto the ToolFlow catalog
• 712 synthesized ABSTAIN examples generated by Qwen3-235B-A22B-Instruct-2507 to cover prompts where no catalog tool applies

The 16% ABSTAIN ratio was the single biggest leverage point in training. v1 of this adapter was trained without ABSTAIN data and scored 4.4% semantic-valid (model never refused). Adding the ABSTAIN cases lifted v2 to 60.4% — a +56 pp improvement, McNemar p ≈ 10⁻⁴². See the project SURPRISES log for the full lesson.

Training data is NOT redistributed in this repo — xlam is gated on Salesforce's HuggingFace page and republishing derivatives openly would be a workaround. The adapter weights are derivative of Qwen3-8B (Apache 2.0) and license-clean.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("s4um1l/toolflow-qwen3-8b-utility-v2")
model = PeftModel.from_pretrained(base, "s4um1l/toolflow-qwen3-8b-utility-v2")
model.eval()

# Build the system prompt with the 20-tool catalog (see tools.json in the
# repo for the canonical list). The training data uses the zero_shot prompt
# template at https://github.com/s4um1l/toolflow/blob/main/prompts/zero_shot.md
SYSTEM = """You are a tool-calling assistant. Given a user request, respond with EXACTLY ONE JSON object specifying which tool to call and arguments. If no listed tool fits the request, respond with {"tool": "ABSTAIN", "arguments": {}}.

Available tools:
- math.area_circle: Calculate the area of a circle given its radius.
  schema: {"type":"object","properties":{"radius":{"type":"number","exclusiveMinimum":0}},"required":["radius"],"additionalProperties":false}
... (full 20-tool catalog)

Output format:
{"tool": "<tool_name>", "arguments": <args_object>}

Do not include explanation, markdown, or text outside the JSON object."""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user",   "content": "What's the area of a circle with radius 4?"},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,  # important: disables Qwen3's thinking mode for direct JSON output
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# {"tool": "math.area_circle", "arguments": {"radius": 4}}

For inference at scale, vLLM + PEFT serving is the recommended path (not benchmarked in this release).

Known limitations

Distribution-shift on formal-style ABSTAIN prompts. Recipe A v2 abstains correctly on 99% of training-distribution prompts, but only 62% of BFCL-style (formal/academic) abstain prompts. The 10 pp gap to Sonnet on the eval is almost entirely this one failure mode (94 of 107 incorrect cases are "should-have-abstained-on-BFCL-stylistic-prompt"). This is not a capability gap — the model knows how to abstain — it's a coverage gap in training data. Fixable in the next iteration by adding ~500 BFCL-style abstain prompts to training.

No vLLM/PEFT serving benchmarks. The "16× cheaper" cost claim is an estimate from token-pricing differential (Sonnet list pricing vs vLLM throughput-mode estimates). Real production deployment may show different numbers; benchmark before relying on this for cost projections.

No catastrophic forgetting evaluation. The adapter wasn't tested against IFEval or general-instruction benchmarks. It might have lost some general capability during training. The evidence available: schema validity at 86%, no JSON parsing failures across 270 eval cases. But a proper ablation isn't done.

Narrow catalog. This was trained for the specific 20-tool ToolFlow utility catalog. Performance on tools outside this catalog is undefined.

Companion model

The 4B variant of this same recipe — same training data, same hyperparameters, half the parameters — is at s4um1l/toolflow-qwen3-4b-utility-v2. It scores 50.4% semantic-valid (10 pp lower) but uses ~30% less inference memory. Pick the 4B if serving constraints make 8B awkward; pick this 8B otherwise.

License

Apache 2.0 (inherited from Qwen3-8B base model). Adapter weights only — no training data redistributed.

Project + reproducibility

• Project repo: https://github.com/s4um1l/toolflow
• Eval set: 270 BFCL prompts re-labeled by Qwen3-235B against the utility catalog. SHA256 0f1508cb…aa8419 in the repo.
• W2 comparison report (with statistical significance): reports/week2_comparison_report.md
• Recipe A v2 detailed report: reports/recipe_a_report.md
• Project tag: v0.2-week2

Citation

@misc{toolflow_qwen3_8b_utility_v2,
  title  = {ToolFlow Qwen3-8B Utility v2 (LoRA adapter)},
  author = {Srivastava, Saumil},
  year   = {2026},
  url    = {https://huggingface.co/s4um1l/toolflow-qwen3-8b-utility-v2}
}

If you use this adapter, please also cite the upstream training data sources (xlam-function-calling-60k and glaive-function-calling-v2) and the Qwen3 base model.