qwen3-4b-tau2-telecom-grpo

Qwen3-4B-Instruct — GRPO on τ²-bench (telecom)

Qwen3-4B-Instruct-2507 trained with GRPO on the telecom domain of τ²-bench, a benchmark for customer-support agents that must follow a domain policy, call backend tools and talk to a (simulated) customer over many turns.

This is a research baseline, not a tuned system.

Results

Scored on the τ²-bench telecom test split (40 tasks), 3 attempts per task (120 episodes), temperature 0.4, up to 100 agent turns per episode. Reward is τ²-bench's own evaluate_simulation — a task counts as solved only when every check (DB state, action, env assertion, NL assertion) passes.

Per-task consistency, out of 40 tasks:

The distribution is bimodal: 35 of 40 tasks are either never solved or always solved. Sampling more therefore buys little (pass@3 is only 6.7 points above avg@3), and the remaining gains have to come from capability on the 25 unsolved tasks — mostly multi-fault repairs needing 7–10 correct actions.

Single-pass validation during training, for reference: 0.250 after epoch 1, 0.300 after epoch 2.

Training

verl / verl-agent (GiGPO), 6×A100-80GB.

Action format

One tool call per turn, Qwen3-native, with respond and done expressed as tools:

<think>reasoning</think>
<tool_call>{"name": "get_customer_by_phone", "arguments": {"phone_number": "555-123-2002"}}</tool_call>

<tool_call>{"name": "respond", "arguments": {"content": "message to the customer"}}</tool_call>
<tool_call>{"name": "done", "arguments": {}}</tool_call>

done matters: τ²-bench only scores simulations that end in AGENT_STOP or USER_STOP. An episode that merely runs out of steps returns reward 0 with every per-check field empty, so it cannot even earn partial credit.

Prompt

The prompt carries a compact telecom policy (~505 tokens, rewritten from τ²-bench's ~5,900-token main_policy.md + tech_support_manual.md), the agent's 13 callable tools as JSON schema, and — separately — the 30 tools that only the customer can run on their phone (check_network_status, toggle_airplane_mode, …).

Splitting those two lists is not cosmetic. τ²-bench's telecom policy describes the device actions in call syntax, and a model that reads it will emit check_network_status() and get "Tool not found". Naming them as customer-operated removed that failure mode entirely.

Reward shaping (training only)

τ²-bench's reward is binary, which leaves GRPO with zero advantage whenever no rollout in a group succeeds — observed as grad_norm collapsing from 2.9 to 0.03 with reward pinned at 0.000. Training therefore added a dense term built from τ²-bench's own per-check detail:

shaped = tau2_reward + 0.4 * partial_score      # telecom

The coefficient stays below 1.0 so a full success always outranks partial progress. All numbers reported above use the unshaped τ²-bench reward.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MANGSEOK123/qwen3-4b-tau2-telecom-grpo"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

The model expects the prompt layout described above (compact policy + tool schemas + customer-device tool list + the action contract). Prompted differently, it will not behave as measured — in particular it is trained to emit exactly one <tool_call> per turn and to call done when finished.

Limitations

• 40 test tasks means wide error bars: ±4.2 points on avg@3, so a 5-point difference against another checkpoint is not distinguishable from noise.
• Trained and evaluated against a gpt-4.1-mini user simulator. A different simulator changes the task.
• telecom only. The retail domain was not trained here.
• 14 optimizer steps is a short run; this is a baseline, not a converged policy.