Qwen3.5-4B-MatRL-MT-RL

Qwen3.5-4B — MatRL multi-turn RL

Multi-turn RL on top of amayuelas/Qwen3.5-4B-MatRL-MT-SFT, for crystal-structure inverse design. The agent proposes candidate structures, evaluates them against a machine-learning interatomic potential, refines, and submits — rewarded on validity, novelty, and thermodynamic stability.

Results

SUN rate (Stable, Unique, Novel) — n=100 prompts × r=8 = 800 rollouts/track. equflashv2 is the reward-path judge and therefore the reference; orb-mpa and eqv3 are held-out judges that never touched training. Judging protocol is independent of the reward path (200 relaxation steps, fmax 0.02).

The base model scores 0% on these tracks under strict scoring — it almost never calls submit (3 times in 2,880 rollouts), so nothing is reconstructed to score. Learning to commit within the turn budget is most of the gap.

On reward hacking

The policy is optimized against equflashv2, so the number that matters is whether judges it never saw agree:

Held-out judges track the reward judge closely and exceed it on the unconditional track. The gains are not artifacts of the training judge.

Training

Trained on 8×A100-40GB.

Known training caveat

Steps ~150–200 ran with 9–15% of each batch clipped at the trainer's 16,384-token window, so those episodes' endings — including the submit turn — contributed no gradient. This checkpoint is released as-is because the held-out evaluation above shows it is not inflated: independent judges match or exceed the reward judge on every track. A truncation-clean redo of the final 50 steps was trained separately and scored lower (13.5 / 12.5 / 10.8%), so the clipping does not appear to have produced a falsely strong model.

Important: thinking channel

Reasoning lives in the native thinking channel, preserved across tool calls. Do not sample with enable_thinking=false — it disables the behavior this model was trained to use.

Serving

Includes preprocessor_config.json / video_preprocessor_config.json — vLLM requires them for this VL-capable model class even for text-only serving.