video-8b-grpo-n16-warm-explore

video-8b-grpo-n16-warm-explore

RLVR post-training of Qwen/Qwen3-VL-8B-Instruct on multiple-choice video QA with fully-async GRPO plus entropy-aware token-dropout exploration, warm-started from the cold-start base RL control at its step 60.

This is the highest offline full-set score of the 8B video campaign: mean_accuracy 0.4942 @ step 40. It edges the previous best (video-8b-grpo-base @ 80, 0.4918) by +0.24 pt, and beats its own paired no-exploration control by +0.63 pt at the same step.

Keeper = global_step_40 (peak).

Offline evaluation

Full set = 5,645 rows: VideoMME-v1 2,700 + Video-Holmes 1,837 + PerceptionTest-comp 1,108. num_failed = 0. mean_accuracy is the unweighted macro mean of the three benchmark accuracies.

Read this before using the number

Three caveats, stated because they change how the result should be interpreted.

1. It is a transient peak, not a plateau the run held. Steps 20 and 40 are 0.4941 and 0.4942, then the run declines monotonically to 0.4622 @ 140. Its paired advantage over the control goes +1.3 pt → −1.8 pt across the same span. Later checkpoints of this run are worse than doing nothing.
2. The step axis is warm-started. Training resumed from the control's global_step_60, so "step 40" is roughly gs100 cumulative. It is not comparable on training budget to video-8b-grpo-base @ 80, which sits on an absolute cold-start axis.
3. The margin is inside the campaign's own noise. The top five checkpoints of this campaign span 1.0 pt (0.4942 / 0.4918 / 0.4913 / 0.4890 / 0.4879) on the same corpus and the same 5,645 rows. Across the wider campaign, exploration on video is best described as a wash — see the sibling cards, which reach the same conclusion from different arms.

Training setup

Sibling checkpoints

Same corpus, same 5,645-row evaluation, directly comparable:

• ngqtrung/video-8b-grpo-base — cold-start plain GRPO, 0.4918 @80
• ngqtrung/video-8b-grpo-ppexplore-n16k8 — 0.4913 @80
• ngqtrung/video-8b-grpo-ppexplore — 0.4890 @140
• ngqtrung/video-8b-grpo-sft770 — OMR-SFT-770 warm start, 0.4845 @180

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "ngqtrung/video-8b-grpo-n16-warm-explore"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")

The checkpoint is a merged FSDP export: weights, tokenizer and processor are all present, so it loads directly in vLLM as well.