qwen3-8b-tau-sft

qwen3-8b-tau-sft

Non-thinking cold-start SFT of Qwen/Qwen3-8B for tau-bench retail, trained on successful synthetic retail trajectories from fuvty/tau-bench-synthetic.

This is the epoch-1 checkpoint, which is the one that passed the pre-RL gates and was used to initialize GRPO. It is a behavior-priming checkpoint: it teaches the model to act as a non-thinking, one-tool-call-per-turn tau-bench agent. It is not a finished tau-bench policy on its own — see Results.

What "non-thinking" means here

The tokenizer/chat template is a patched Qwen3 template whose assistant branch renders <|im_start|>assistant\n<think>\n\n</think>\n\n + content verbatim, and whose generation prompt ends with that same empty-think prefill. Because </think> is therefore prompt-side, think-relapse is structurally impossible even at temperature 1.0. The patched chat_template.jinja and the embedded copy in tokenizer_config.json ship with this repo, so apply_chat_template reproduces the training format exactly.

Tool calls are emitted as inline Hermes text in the assistant content (the patched template does not render structured tool_calls lists):

{optional user-visible text}
<tool_call>
{"name": "<fn>", "arguments": {<args object>}}
</tool_call>

At most one call per turn. Tool results are supplied as role: "tool" messages, which the template wraps in <tool_response>…</tool_response> inside a user turn.

Training data

795 reward-1.0 retail trajectories from traj-GLM5 were filtered down to 595 rows over 98 tasks:

• airline dropped entirely (it is a held-out zero-shot transfer eval)
• 12 of 180 synthetic retail tasks dropped by a contamination gate that compares write-action goal signatures against the shipped tau-bench retail task sets. All 12 were retail-train collisions; zero collided with retail-test
• 135 trajectories dropped for containing a turn with more than one tool call
• reasoning traces (reasoning_content) stripped; the tau2 greeting turn dropped

Loss falls on exactly assistant content + <|im_end|>. The assistant prefill header, the inter-turn newline and every observation block are present but masked out, so the training token stream is byte-identical to what an inference server is fed.

Training

Final loss 0.118 at epoch 3; this epoch-1 checkpoint was selected because it already passed the boundary-token gate below and further epochs were not needed for priming.

Gates

Boundary-token probe (does SFT actually train the turn-ending specials?):

Format/geometry gate: converted rows were replayed through the actual multi-turn rollout code with the row's turns scripted as sampled outputs, asserting exact-token prefix alignment of the generation prompt at every turn, loss-token equivalence, tool-parser round-trip, and eos invariants — 595/595 passed. Longest assistant turn: 613 tokens.

Results

Evaluated with greedy decoding on tau-bench retail-test (115 tasks) and airline (50 tasks, never trained on), with a gpt-4.1-mini user simulator.

Sampled (T=1.0, 4 samples/task) after those 32 GRPO steps, where the decoding pathology is absent:

Read the SFT-only number carefully. 0.043 greatly understates the policy: under sampling (T=1.0) the same weights score 0.637 average success on retail-train with pass@16 ≈ 0.97. The gap is a greedy-decoding pathology — 93% of greedy eval episodes exhaust their 30-turn budget in a repetition loop rather than terminating. Subsequent GRPO mostly repaired that pathology (greedy truncation 0.93 → 0.33, repetition 0.48 → 0.15) rather than adding raw capability, which is why greedy retail-test climbs 7.5× while sampled training reward barely moves.

So: this checkpoint is a strong sampled tool-use policy and a weak greedy one. Use sampling, or RL from it.

Intended use and limitations

Research artifact for tau-bench agent RL. Not a general assistant, not tuned for safety or chat. Its system prompt at training time was the synthetic dataset's own <instructions>/<policy> wrapper and its tool schemas were the dataset's tau2-style retail variant, which differ in three tool names from tau-bench's own retail toolset (cancel_order vs cancel_pending_order, find_user_by_contact vs find_user_id_by_email, find_user_by_name vs find_user_id_by_name_zip). In practice the model conditions on whatever schemas are given in context and transferred to the real tau-bench names with essentially no penalty (5 parse failures across 165 eval episodes).

License

Apache-2.0, inherited from both Qwen3-8B and the tau-bench-synthetic dataset.