toolflow-qwen3-4b-utility-v2

ToolFlow Qwen3-4B Utility — v2 (LoRA adapter)

A QLoRA adapter on top of Qwen/Qwen3-4B, fine-tuned for tool-calling against a 20-tool utility catalog. The smaller-model variant of the ToolFlow project — same recipe and training data as the 8B toolflow-qwen3-8b-utility-v2, trained to answer one specific question: is a 4B model good enough on this narrow tool-calling task at half the parameter footprint?

Headline results

Evaluated on the same 270-case eval set as the 8B variant (BFCL v3 prompts re-labeled against the utility catalog by Qwen3-235B):

Statistical significance vs the 8B variant: Δ semantic-valid = -10.0 pp, 95% CI [-15.2, -5.2], McNemar p = 0.0003. The gap is real, not noise.

When to pick this 4B over the 8B variant

For most cloud-serving production use, prefer the 8B variant — the 10 pp quality gap doesn't justify the ~$16/mo savings at 100K tasks/mo. The 4B becomes the right pick when serving infrastructure is genuinely constrained (memory or hardware compatibility), not when per-task cost is the metric.

What this adapter does

Same as the 8B variant — given a user request and a system prompt enumerating the 20-tool catalog, emits a JSON tool call (or {"tool": "ABSTAIN", "arguments": {}} if no tool fits).

Training details

Training data

Identical to the 8B variant: 4,461 examples (3,749 positive tool-calls from xlam + Glaive, plus 712 synthesized ABSTAIN cases via Qwen3-235B). See the 8B model card or the project README for the full provenance and licensing.

Per-category breakdown (vs 8B)

The 4B is uniformly weaker (8-15 pp drop everywhere); the gap doesn't concentrate in one category. The biggest gap is multi_call, where sustained context + multiple correct args is harder for the smaller model.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("s4um1l/toolflow-qwen3-4b-utility-v2")
model = PeftModel.from_pretrained(base, "s4um1l/toolflow-qwen3-4b-utility-v2")
model.eval()

# (Build SYSTEM prompt with the 20-tool catalog from the repo, see 8B model card for an example)
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user",   "content": "What's the area of a circle with radius 4?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Known limitations

Same distribution-shift issue as the 8B variant — formal/academic-style abstain prompts (BFCL irrelevance category) are under-served by the xlam-shaped training data. Fixable in next iteration by adding ~500 BFCL-style abstain prompts.

The 4B's gap is uniform across categories rather than concentrated in one failure mode, so improvements need to lift overall capability, not just patch one weakness.

License

Apache 2.0 (inherited from Qwen3-4B base model). Adapter weights only — no training data redistributed.

Project + reproducibility

• Project repo: https://github.com/s4um1l/toolflow
• Recipe B detailed report: reports/recipe_b_report.md
• W2 comparison report: reports/week2_comparison_report.md
• Project tag: v0.2-week2

Citation

@misc{toolflow_qwen3_4b_utility_v2,
  title  = {ToolFlow Qwen3-4B Utility v2 (LoRA adapter)},
  author = {Srivastava, Saumil},
  year   = {2026},
  url    = {https://huggingface.co/s4um1l/toolflow-qwen3-4b-utility-v2}
}