qwen3-4b-tau2-telecom-compsub-1ep

Qwen3-4B-Instruct-2507 — tau2-bench telecom, composition + substitution pairs

Qwen3-4B-Instruct-2507 consolidated with OEL on the tau2-bench telecom domain, using 81 externally authored task–memory pairs: the composition (59) and substitution (22) subsets of telecom_final_pairs, with the 19 recall pairs dropped.

Related runs on the same domain and evaluation protocol: telecom-ours-1ep (all 100 pairs) and telecom-oel-1ep (74 self-extracted experiences).

Training

Hyperparameters copied from the retail memory_aug_final run — the one configuration in this series that scored above its base model — so that the dataset is the only thing that differs from that reference.

The student replays each task without memory; the teacher is the same weights with that task's memory in the system prompt. Only the prompt differs, and no reward is used.

Training trace

Each step reads a different batch, so the loss column tracks batch difficulty rather than convergence. Note that the gradient norm sits well above the 1.0 clip on every step — telecom episodes are long, so the effective step size here is set by the clip rather than by the learning rate. Entropy stays healthy through the final step, unlike sibling runs whose last batch was much smaller.

Evaluation

Not evaluated. This checkpoint was pushed straight after training. For context, on tau2-bench telecom test (40 tasks, 4 trials, no memory at eval) the base model scores avg 0.056 / pass@4 0.175, the 100-pair run scores 0.062 / 0.175, and the self-extracted run scores 0.081 / 0.175 — all within about 1.5 SE of each other.

Known caveat. One of the 40 test tasks appears in this training set. A substitution that removes a fault from a training task reproduced a test task's exact environment state (break_apn_settings + lock_sim_card_pin); the task ids still carry the training signature, so only an initial_state comparison reveals it.

Usage

vllm serve MANGSEOK123/qwen3-4b-tau2-telecom-compsub-1ep \
  --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 40960