TT-OPD-Qwen3.5-9B

TT-OPD-Qwen3.5-9B

TT-OPD-Qwen3.5-9B is a Qwen3.5-9B checkpoint post-trained with multi-turn agentic reinforcement learning on medical tasks, using turn-level on-policy self-distillation (TT-OPD) on top of GRPO.

This card documents only what we changed — the training direction, the objective, and the behaviour we measured. Everything about the underlying architecture, tokenizer, multilingual coverage and general capability comes unchanged from Qwen/Qwen3.5-9B; read that card for the base model's properties and benchmarks.

---

What it was trained to do

The model is trained as a medical agent, not as a question answerer:

• Each episode is a clinical or biomedical task solved over multiple tool-use turns against a medical tool environment (literature/evidence search, knowledge lookup, structured record queries) with a final submit_answer call.
• Reward is on the outcome of the episode, so the policy is optimised for reaching a committed answer through the tool loop, not for producing fluent prose.

The single clearest behavioural change is answer commitment. On the same harness, the rate of episodes that end without ever producing an answer drops sharply:

Objective

TT-OPD = turn-level on-policy distillation, added to GRPO. As configured in this run:

• Gradient-free EMA teacher — the teacher is an exponential moving average of the student itself, so there is no external teacher model and no separate teacher training.
• Bidirectional KL between student and EMA teacher, applied with a sign flip so that correct trajectories are stabilised and incorrect ones are actively pushed away.
• Turn-level span — the KL covers the whole trajectory (all turns), not just the final response, which is what makes the regulariser act on the agent loop rather than on one answer.
• Top-K position filtering — the distillation term is applied only at the highest-signal token positions.
• Cosine length shaping on the reward.

Released weights are step 660, the end of the run.

Measured behaviour

Agentic multi-turn evaluation on our own harness (5-turn budget, tools + knowledge base), 3 independent runs per cell, mean ± sd. The base row is the same Qwen3.5-9B run through the identical harness, so this is a controlled comparison, not a comparison against published single-turn numbers.

Read this honestly:

• MedQA +3.0 pp over base is the one solid gain, and most of it is the answer-commitment effect above rather than better per-answer reasoning.
• MMLU-Med +0.8 pp is inside ~1.5 sd — treat it as "no regression", not as a gain. It is worth stating plainly that this checkpoint shows no catastrophic forgetting of parametric medical knowledge on this harness.

Degeneration checks

Measured over 1,273 episodes / 6,334 generated turns, against the base model on the same harness:

No repetition collapse, no length runaway, no answer-option collapse, and no tool-call format degradation (99.5% of turns emit a well-formed tool call).

Known limitations

• It is a tool-using agent. It was trained and evaluated entirely inside a multi-turn tool loop that ends in a submit_answer call. We have not measured it as a plain single-turn chat model without tools, and we would not expect the numbers above to carry over to that setting. Serve it with its tool loop.
• Short answers on open-ended questions. On free-text medical questions the median answer this model submits is about 55 characters, against about 196 for the base model on the same prompts. The training signal rewards committing an answer, and the model has specialised toward short, submittable answers. Do not use it for long-form patient-facing explanation.
• Retrieval is not the source of the gain. The gain here should be read as agent-loop behaviour (commitment, format, turn use), not as evidence of learned retrieval-augmented reasoning.
• English, medical/biomedical domain only. Behaviour outside that distribution is the base model's.
• Not a clinical tool. This is a research checkpoint. It must not be used for diagnosis, treatment decisions, or any patient-facing purpose.

Usage

from transformers import AutoModelForCausalLM, AutoProcessor

model = AutoModelForCausalLM.from_pretrained(
    "Minbyul/TT-OPD-Qwen3.5-9B",
    torch_dtype="bfloat16", device_map="auto", trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(
    "Minbyul/TT-OPD-Qwen3.5-9B", trust_remote_code=True,
)

Serving (SGLang, matching how it was evaluated):

python3 -m sglang.launch_server \
  --model-path Minbyul/TT-OPD-Qwen3.5-9B \
  --served-model-name ttopd-9b \
  --host 0.0.0.0 --port 30000 --tp-size 1 \
  --context-length 131072 --mem-fraction-static 0.85 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --trust-remote-code

The checkpoint keeps the base model's vision tower (model.visual.*, 333 tensors), so the composite Qwen3_5ForConditionalGeneration config loads as-is. RL training was text-only; the vision path is inherited from the base and was not trained here.

Citation

@misc{ttopd_qwen35_9b,
  title  = {TT-OPD-Qwen3.5-9B: turn-level on-policy distillation for multi-turn medical agents},
  author = {Jeong, Minbyul},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Minbyul/TT-OPD-Qwen3.5-9B}}
}

License

Apache-2.0, inherited from the base model (see LICENSE).