Downloads · 30 days
14
3% of all-time downloads
yuhan-nlp/verl-grpo-medium-qwen3-4b-step50
verl-grpo-medium-qwen3-4b-step50 is a text generation model from yuhan-nlp. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
Qwen/Qwen3-4B trained with GRPO on the CollabLLM medium document-writing task using the verl CollabLLM recipe (no SFT warm start). This is a merged HF checkpoint — load it directly, no adapter step.
Downloads · 30 days
14
3% of all-time downloads
All-time downloads
418
Public
Parameters
4.4B
8.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8.8 GB · 100%
From the Hugging Face model README
Qwen/Qwen3-4B trained with GRPO on the CollabLLM medium document-writing task using the
verl CollabLLM recipe (no SFT warm start). This is a
merged HF checkpoint — load it directly, no adapter step.
This checkpoint: an intermediate point: BLEU essentially flat vs base (t=+0.22); the token term is already dropping.
Three roles, thinking off everywhere: Qwen3-4B assistant (this model),
Qwen/Qwen3.5-9B user simulator, Qwen/Qwen3.5-27B judge. Flags:
--user_sim_prompt sim_gap_paper_lazy_full --assistant_max_tokens 4096 --max_new_turns 14.
| condition | BLEU | interactivity | tokens (k) | MR |
|---|---|---|---|---|
base Qwen3-4B (no proact) | 0.4491 | 0.831 | 2.935 | 0.9870 |
| GRPO step 50 | 0.4524 | 0.819 | 2.281 | 1.0433 |
| GRPO step 100 | 0.4827 | 0.801 | 2.077 | 1.0761 |
| GRPO step 129 | 0.4825 | 0.744 | 1.944 | 1.0316 |
| proact prompting (no training) | 0.4669 | 0.825 | 1.678 | 1.1236 |
| ← this model (step 50) | 0.4524 | 0.819 | 2.281 | 1.0433 |
critic/rewards/mean — it rises through
step 129, because the reward credits brevity directly while the benchmark judge penalises the
interactivity that brevity costs.GRPO, train_batch_size=16, rollout.n=8, lr=1e-6, kl_loss_coef=0.001 (low-var KL),
8 GPUs (FSDP, world_size_8), reward = bleu_score (+1) + interactivity (+1) +
token_amount (−0.1). Reference run: wandb photon/verlxcollabllm/e1adivlo, val@0 = 1.0618.
Full setup, the self-contained docker image, and the two acceptance tests are documented in
studynotes/new_cluster_onboarding.md of the code repo.