Qwen3.5-9B-sdpo-react-mathcodesearch-grpo-arm-e-step29

Qwen3.5-9B-sdpo-react-mathcodesearch-grpo-arm-e-step29

RL post-training of Qwen/Qwen3.5-9B on a multi-turn, native-tool-calling (ReAct-style) mixture of math + code + search tasks, using GRPO with an SDPO self-skill objective ("arm e").

This is the step-29 checkpoint, which is the peak of the training curve — see Why step 29 below.

[!IMPORTANT] Language model only. The conversion from the training checkpoint exports the 427 language-model tensors and drops the base model's vision tower (333 tensors) and MTP heads (15 tensors), because only the language model was trained. config.json still declares Qwen3_5ForConditionalGeneration for compatibility with the base tokenizer/config, but image input will not work. Text generation loads and serves normally under Transformers / vLLM / SGLang.

Training setup

Held-out results

pass@1 / pass@8, greedy-free sampling, 16384-token response cap:

AMO-Bench and OJBench are the best numbers we have on record for a 9B model in this line of work (previous best: AMO 22.7, OJBench 29.5).

Caveats worth knowing before you compare against these:

• AIME 2024/2025 are saturated for this model — pass@8 reaches 100%, so they can no longer rank checkpoints. Our training curve uses AIME-2026 instead.
• AMO-Bench pass@8 is depressed by truncation, not by reasoning failure: the median sample sits exactly at the 16384-token cap and 18.5% of samples are truncated. A larger generation budget should raise it.
• No tool-use collapse: on OJBench only 15.7% of rollouts answer without calling a tool, versus 74.0% for the base model, and repetition rate is 0.

Why step 29

Two independent runs of this arm agree to within 0.6pp at every shared eval step, and both peak at step 29. Aggregate held-out pass@1 by step:

Past step 29 performance decays monotonically, led by code (LiveCodeBench-v6-functional 63.3 → 59.1) with repetition rate climbing 0 → 0.6 → 4.2%. Training this arm longer is actively worse, so 31 rollouts is the right budget and step 29 is the checkpoint to use.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ipfipfipf/Qwen3.5-9B-sdpo-react-mathcodesearch-grpo-arm-e-step29"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")

The model was trained with thinking enabled and with native tool calling, so serve it with the shipped chat_template.jinja and pass tools through the template's tools argument. Under SGLang, use the qwen3_coder tool-call parser (Qwen3.5 emits XML-style <function=...> calls, not Qwen2.5-style JSON).