vigorl-ckpts

ViGoRL V* multi-turn RL — 3B ablation checkpoints (seed 42)

Reproduction of ViGoRL (arXiv:2505.23678) multi-turn GRPO on V* (visual search) with five rollout-time ablations, all from the same SFT init and seed. One folder per ablation:

Training

• Base: gsarch/ViGoRL-Multiturn-MCTS-SFT-3b-Visual-Search, verl/EasyR1 multi-turn GRPO, vLLM 0.8.3 rollout.
• Data: vigorl_SA_RL (V*), seed 42, 8 GPUs, rollout batch 128 x n=8, global batch 64, max response 4096, crop 672, offset 182, LR 1e-6 constant after 5% warmup, KL 1e-2, max grad norm 0.2, vision tower frozen.
• Stopped at step ~250 of the 500-step schedule (250 for all; a few ran to 258-288 before stopping).
• vLLM prefix caching disabled. The V1 engine enables it by default and it corrupts multi-turn rollouts that append image crops between turns; with it on, tool calling degrades badly at step 0.
• Rollout-format fixes relative to the upstream branch: newline before </observation>, the V* system prompt made byte-identical to the MCTS-SFT prompt, and the forced-answer string masked out of the loss.
• Code: williamium3000/demystify-twi, branch vigorl, commit cc1882a.

Training metrics at the stopping point

Format reward ceiling is 1.60 (well-formed rollout + four distinct search coordinates).

All five tool-calling ablations sit at the ceiling, i.e. essentially every rollout issues four valid, distinct search calls.

V*Bench (191 items, step 25 only)

Evaluated with eval_vstar.py (HF Transformers, 4 search turns, temperature 0.5, single sample). These models answer with MCQ letters, because the training reward scores letters, so plain string match against the option text undercounts them; the letter-aware column maps the letter back to its option.

At step 25 every ablation is still below the SFT init. Step-250 evaluations are not done yet, so the uploaded checkpoints are not benchmarked; train-time accuracy rose from ~0.45 to ~0.77 after step 25.