Downloads · 30 days
11
13% of all-time downloads
HumorR1/policy-qwen3vl-2b-grpo-newyorker
policy-qwen3vl-2b-grpo-newyorker is a machine learning model from HumorR1. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
A captioning policy for New Yorker–style cartoons trained in two stages:
Downloads · 30 days
11
13% of all-time downloads
All-time downloads
82
Public
Repo size
406 MB
Likes
0
Public
Click a slice to open those files.
.safetensors197 MB · 89%
From the Hugging Face model README
A captioning policy for New Yorker–style cartoons trained in two stages:
HumorR1/rm-qwen25vl-3b-20k).
Reward = sigmoid(RM_score) for completions matching <caption>...</caption>,
else 0.The output format is <think>...brief reasoning...</think>\n\n<caption>X</caption>.
50 cartoons × 5 samples = 235 generations per policy. Eval uses
max_new_tokens=8192 — the chain-of-thought is verbose (median ~2500
tokens), so smaller budgets clip most completions and make the metric
meaningless. Use this same budget at inference.
| base | this policy | |
|---|---|---|
| emitted-caption RM mean | -2.36 ± 0.09 | -2.17 ± 0.07 |
| format compliance | 0.889 | 0.962 |
| no-caption due to truncation | 3 | 0 |
| Δ vs base on emitted RM | — | +0.19 ± 0.11 SE = +1.69σ |
The SFT-only intermediate (no thinking) scored emitted RM -1.74 (stronger raw quality, but no chain-of-thought). This policy keeps the thinking format the project proposal calls for and beats base by +1.69σ on emitted RM while completing captions ~7 pp more reliably (96% vs 89%).
scripts/synthesize_thinking.py — base Qwen3-VL-2B-Thinking generates
a brief explanation for each (cartoon, top-rated caption) pair. Saved
as data/caption_sft_train_with_thinking.scripts/train_sft.py — SFT for 2 epochs on the augmented dataset.
LoRA r=α=32, lr=2e-4 constant, no warmup, eff_bs=32, bf16, FA2.
Target = {thinking}</think>\n\n<caption>{caption}</caption>.qwen3vl-2b-sft-think-merged.scripts/train_grpo_qwen3vl.py with POLICY_MODEL_NAME=...sft-think-merged,
FORMAT_BONUS=0.0, MAX_STEPS=200, NUM_GENERATIONS=8,
MAX_COMPLETION_LENGTH=1536. Single A100, vLLM colocate.Held-out cartoons (RM scores in parentheses, higher is better):
Contest 543 (cartoon: elephant in a living room; people ignore it)
- base (-4.66): "The two people in the living room try to read while the elephant stands there, like it's the secret probably lost."
- policy (-0.89): "Elephant's in the living room, but the humans are still reading."
Contest 568 (cartoon: woman in glass talks to two people stuck in buckets)
- base (-3.27): "At the party, you're supposed to sip, but you're bound in buckets—no toast, just a bucket-and-meeting!"
- policy (-0.90): "At the party where the only social event that works is standing in buckets."
stop=["</caption>"] and rely on
best-of-N sampling at inference — best-of-3 should hit a valid caption
in >70% of cartoons.For best throughput, vLLM with enable_lora=True and a LoRARequest
pointing at this adapter (see scripts/eval_policy.py in the source repo).