qwen3-8b-tool-recovery

qwen3-8b-tool-recovery

Qwen3-8B fine-tuned for tool-failure recovery: when a tool/API call fails (server error, malformed response, timeout), this model detects the failure and takes corrective action — retries the call, switches to an alternative tool, asks the user a concrete question, or reports the failure honestly instead of hallucinating a result.

This is the research artifact behind the QUENCH study: "Do trained agentic skills survive 4-bit quantization?" Answer: yes — this model's recovery skill is statistically unchanged under NVFP4 (see table below). Quantized variants: NVFP4 · FP8-dynamic.

Results

Recovery Rate (RR) = share of injected-failure trials with full recovery (0–3 rubric, scored 3 by GLM-5.2 judge; 60 BFCL v3 tasks × 3 failure types; task-level bootstrap 95% CI). Both variants quantized with the same llm-compressor recipe (no cross-recipe confound).

• Trained recovery survives NVFP4: Δ = -0.6pp [-8.3,+6.7] vs BF16 (not significant).
• Same NVFP4 quantization costs ~3pp MMLU in both variants — the trained behavior is more quant-robust than parametric knowledge.
• General-capability guard: MMLU-400 = 71.9% (vs base 61.9% under identical prompting — the SFT also improved answer-format compliance; no catastrophic forgetting).
• Known failure mode: after a successful retry that returns data-free confirmation, the model may state specifics not present in the tool result (flagged in 40–46% of recovery trials by a faithfulness-strict judge). Behavioral-recovery vs faithfulness-conditioned readings are both reported in the eval transcripts.

Training

• Base: Qwen/Qwen3-8B · LoRA r=32, α=64, 2 epochs, lr 2e-4, seed 300, assistant-only loss (~9 min on 1× DGX Spark GB10); merged to bf16.
• Data: 682 synthetic trajectories — 480 recovery (4 strategies × 3 failure types × 8 tool domains), 152 plain tool-success, 50 no-tool examples. Failure texts are disjoint from the eval set's injected failures.
• Template note: training text is rendered with the exact inference chat template, with the empty <think> block normalized onto every assistant turn (enable_thinking=False asymmetry between training render and inference prefill otherwise silently breaks tool-calling in Qwen3 SFT).

Usage (vLLM)

vllm serve GurkanOz/qwen3-8b-tool-recovery \
  --enable-auto-tool-choice --tool-call-parser hermes

Use with chat_template_kwargs: {"enable_thinking": false} and OpenAI-style tools. On tool errors, return the error payload as the tool result — the model will retry/switch/ask instead of hallucinating.

Limitations

Research artifact, not a production assistant. English-only training data across 8 synthetic tool domains; single seed; evaluated on BFCL single-turn tasks with injected failures. The retry environment used for evaluation returns synthetic success payloads. Inherits Qwen3-8B's base limitations and license (Apache-2.0).

Citation

@misc{ozkan2026quench,
  title  = {QUENCH: Trained Agentic Skills Survive 4-Bit Quantization},
  author = {Özkan, Gürkan},
  year   = {2026},
  url    = {https://huggingface.co/GurkanOz/qwen3-8b-tool-recovery}
}