qwen35-4b-videomme-grpo

Qwen3.5-4B — GRPO/DAPO on Video-MME-v2 (800 steps, 100 frames)

LoRA adapter (r=16, α=32, language side only) trained with GRPO on Video-MME-v2 multiple-choice video QA. Training code: https://github.com/infinitylogesh/videomme-rl

Results — 400 held-out questions, 100 frames, thinking mode

Held-out = 100 videos disjoint from the 500 training videos (split by video, never by question: each video carries 4 questions, and the logic groups chain them). Every run uses the same seed and the same option permutation, so all comparisons are paired.

Chance is 0.129 (8 options). Paired McNemar, strict:

Report STRICT accuracy, not loose

strict = the model emitted <answer>X</answer> and X was right. A "loose" parser that also accepts the last stray A–H letter in the text inflates the baseline enormously: of its 79 loose-correct answers, only 25 had a real answer tag — 53 came from completions that hit the token cap without answering at all. With 8 options that is a 1-in-8 lottery, awarded on 77% of questions. The two metrics converge for the trained model (0.3050 vs 0.3050) and diverge for the baseline (0.1975 vs 0.0625). Loose accuracy rewards exactly the behaviour this training removes.

What the training actually changed

The base model does not fail by being wrong so much as by never finishing: it reasons until the token cap and emits no answer. Truncation went 0.770 → 0.0025, answer-tagging 0.223 → 0.9975. It still reasons — median completion is ~300–500 tokens with substantive analysis — it just concludes now.

Config

lr 1e-5 · beta 0.0 (no KL) · loss_type dapo · scale_rewards none (Dr.GRPO)
epsilon_high 0.28 (DAPO Clip-Higher) · num_generations 8 · 100 frames @ max_pixels 50176
max_completion_length 2048 · thinking mode ON · reward 0.9*correct + 0.1*tagged, strict

Reward = 0 for a truncated completion, and truncated rollouts stay in the loss (deliberately not mask_truncated_completions) — that is the only pressure against rambling.

Negative results worth keeping

• Frames are not the lever. 32 → 100 frames moved the baseline only +2.25pp (p=0.41). Even on fine-grained temporal questions ("how many times…", "first 15 seconds") it went 0.054 → 0.107, still below the 0.129 chance floor. The gains above are the algorithm/lr changes.
• Entropy regularisation is the wrong tool here. ~46% of groups have zero reward variance, but measured entropy is 0.55 nats — not a collapsed policy. With entropy_target=0.2 TRL never applied the bonus (entropy_coef = 0.0000 for 400 steps). The zero-variance groups are saturated prompts (always wrong), not an exploration failure. DAPO dynamic sampling would recover the wasted compute but would not make hard prompts solvable — and it is absent from TRL 1.8.
• A within-run reward plateau does not mean no headroom. Reward looked flat at the end of the 400-step run, yet the next 400 steps bought +5.75pp (p=0.02). Judge by eval, not by the reward curve.

Files

• adapter/ — LoRA + optimizer state (resumable)
• results/ — per-question eval jsonl for every run above, plus the exact train/eval splits