video-8b-grpo-ppexplore-n16k8

Qwen3-VL-8B · Video-MC · GRPO + ppexplore (τ0.8, n16/k8 — group-size ablation)

RLVR post-training of Qwen/Qwen3-VL-8B-Instruct on multiple-choice video QA with fully-async GRPO and entropy-aware token-dropout exploration ("ppexplore", τ=0.8) at the rollout stage, using bigger GRPO groups (rollout.n=16, k_explore=8). Warm-started from the SFT-770-RL global_step_60 checkpoint. This is experiment_name=grpo_video_4node_full_v3_24f100k_8b_sft770_rl60_ppexplore_tau08_n16k8_v1, global_step_80 (the keeper).

Group-size ablation — also a WASH: bigger GRPO groups (n16/k8) peak at offline full-set val mean_accuracy 0.4913 @ step 80, then clean-fade. This ties the best-prior keeper (cold-base 0.4918) but does not break the ~0.485 video dead-heat. GRPO group size is not the missing lever — this run closes the 8B video group-size question (7-way dead-heat overall).

Results

Offline full-set eval (VideoMME-v1 2700 + PerceptionComp 1108 + Video-Holmes 1837 = 5645 rows), scored with the repo's vero compute_score. mean = macro-mean of the 3 bench accuracies.

Keeper = global_step_80 (peak):

Trajectory (mean): 0.488 @20 → 0.490 @40 → 0.487 @60 → 0.491 @80 → clean monotone fade (0.481 @120, 0.473 @140), no collapse. (Step 100 eval skipped — reproducibly hung on one deterministic shard, step-100-specific not systemic.) Peak edges the n8 explore siblings by ~0.3 pt = within noise.

Training

• Base model: Qwen/Qwen3-VL-8B-Instruct, warm-started from the SFT-770-RL run's global_step_60 (merged HF _merged/sft770_perf_gs60_hf, itself val 0.4814). Lineage: stock 8B → OMR-SFT-770 → video-RL gs60 → this run.
• Framework: fork of volcengine/verl — ngquangtrung57/verl@videorl-mods. Fully-async GRPO: FSDP2 trainer + vLLM rollouter.
• Reward: dapo-style score = 0.8·accuracy + 0.2·format (FORMAT_WEIGHT=0.2, FORMAT_MIN_THINK_CHARS=100). No KL penalty.
• Topology: 4-node 2+2 — 2 trainer nodes (16-GPU FSDP2, dp=16) + 2 rollout nodes (16 GPU, vLLM TP=2 → 8 replicas). H100×8 per node.
• Batch: ppo_mini_batch_size=16 × require_batches=4 × rollout.n=16 = 1024 rollouts/fit-step (vs 512 at n=8; bigger GRPO groups = 8 explore + 8 anchor per selected prompt).
• Data: GROUP_VIDEO_TRAIN_MC_24F100K (5 video-MC parquets, 24 frames / 100k pixels).
• Optim / seq: lr 1e-6, warmup 25 steps; total_epochs=2; clip_ratio 0.2 / 0.3 (clip_c=10.0); max_prompt_length=17408, max_response_length=16384; enforce_eager=true; gpu_memory_utilization=0.75; staleness 0.4. No OOM at the 256-traj actor update (token-bounded micro-batch held).
• Exploration block (entropy-aware token-dropout at rollout; τ=0.8 with larger groups):

• Validation: inline val OFF (test_freq=10000); video val is offline full-set eval on a dedicated H100 node (vLLM TP1), every 20 fit-steps.
• Train metrics: ~445 s/step (vs ~290 at n=8 — the bigger groups cost ~55% more wall-clock); reward → ~0.88; response_length ~250 tok; format ~1.0; no OOM/collapse. Stopped by user at fit-step ~106/781 (verdict clear).

W&B

Project verl_fully_async (entity quangtrung5705-nanyang-technological-university-singapore). The logical run has three crash-resume segments (vtlkaq8u, iwwcf7do, hb4vi5h3); the last/most-complete (hb4vi5h3) carries the curated label. Train metrics only — video val is offline, not on W&B: https://wandb.ai/quangtrung5705-nanyang-technological-university-singapore/verl_fully_async/runs/hb4vi5h3

Intended use / limitations

Research checkpoint — the group-size ablation arm of an 8B video study. Headline finding: bigger GRPO groups (n16/k8) do not break the ~0.485 video dead-heat (peak 0.4913 ties the best keeper then fades). With this run, seven levers (SFT warmstart, cold-base RL, cold exploration, base-warm explore, SFT-warm explore τ0.95, SFT-warm explore τ0.8, and this τ0.8 n16/k8) all peak ~0.485-0.49 — the video ceiling is bound by something outside the exploration / warm-start / RL-duration / τ / group-size space. Multiple-choice video QA, <think>…</think> then-answer format. No safety/RLHF alignment beyond the base.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "ngqtrung/video-8b-grpo-ppexplore-n16k8"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": "clip.mp4"},
        {"type": "text", "text": "Answer the multiple-choice question. Reason inside <think>...</think>, then give the final letter."},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Citation / lineage

• Base model: Qwen3-VL-8B-Instruct (Qwen team). Inherits the Qwen3-VL license — review the base model's terms; the Apache-2.0 tag refers to this repo's RLVR training artifacts.
• Warm start: SFT-770-RL global_step_60 (lineage: stock 8B → OMR-SFT-770 → video-RL gs60).
• Framework: verl (volcengine/verl), fork ngquangtrung57/verl@videorl-mods; fully-async GRPO (FSDP2 + vLLM).
• Method: entropy-aware token-dropout exploration ("ppexplore", τ=0.8) with enlarged GRPO groups (n=16, k_explore=8). Group-size ablation in a controlled OMR/Video exploration study on Qwen3-VL-8B (docs/experiments_summary_8b.md).