openr1-gemma3-4b-gt-mllm-repro

openr1 x Gemma-3-4B x GT (stock GRPO)

Checkpoints from a full 1-epoch run, saved every 20 steps, so the whole training trajectory can be re-evaluated rather than only its endpoint. Trained with mllm-repro on lmms-lab/multimodal-open-r1-8k-verified.

Weights only -- ZeRO-3 optimizer shards are stripped, so these are for evaluation and further fine-tuning, not for resuming this run.

What this run shows

The in-loop curve is flat. It starts at 0.329 and ends at 0.362, and every one of the 48 evaluations falls in 0.296-0.434 with no trend. Compare the same recipe on Qwen2.5-VL-3B, which climbs 0.283 -> 0.55 within 250 steps: that one has a shape, this one does not. GT training on this dataset produced no measurable gain for this model.

The maximum at step 760 is 0.06 above its neighbours (740 scores 0.375, 780 scores 0.382) while a single measurement on 150 problems carries a standard error of about 0.041, so that peak is noise rather than a checkpoint worth selecting.

Setup

Maximum 0.4342 at step 760. The eval set is 150 problems, so one problem is 0.67 points and a single measurement carries a standard error of roughly 0.041; score candidates on a real benchmark suite before picking one.

In-loop eval (MathVista-150)