Downloads · 30 days
8
14% of all-time downloads
Occupying-Mars/qwen3-4b-symbolic-vanilla-ppo-v1
qwen3-4b-symbolic-vanilla-ppo-v1 is a reinforcement learning model from Occupying-Mars. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
Qwen/Qwen3-4B-Instruct-2507 trained with vanilla PPO (Schulman et al. 2017 — clipped surrogate, learned value head, trainer-side GAE) on the deterministic symbolic tool-calling benchmark from the GRPO-vs-PPO-under-rol…
Downloads · 30 days
8
14% of all-time downloads
All-time downloads
59
Public
Parameters
4B
8.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8 GB · 100%
From the Hugging Face model README
Qwen/Qwen3-4B-Instruct-2507 trained with vanilla PPO (Schulman et al. 2017 — clipped surrogate, learned value head, trainer-side GAE) on the deterministic symbolic tool-calling benchmark from the GRPO-vs-PPO-under-rollout-compaction study (prime-rl fork: ChinmayK0607/blog-rl, branch feat/vanilla-ppo).
This is the uncompacted real-PPO baseline that study was missing: its compacted PPO (rollout-reward critic targets, no GAE) collapsed to 0.25/0.14 pass@4; this run shows actual PPO nearly matches GRPO.
| step | 10 | 60 | 80-100 | 140 | 150 |
|---|---|---|---|---|---|
| val reward | 0.556 | 0.764 | 0.000 | 0.903 | 0.972 |
Reference (same env, Phase-A regime, 8xH100): full GRPO 1.000, compacted GRPO 0.986, segment-normalized GRPO 0.986.
Notable dynamics: entropy collapse around step 80 made a corrupted tool-call token the greedy argmax (val 0 for ~20 steps, 2-turn episodes) — the policy self-recovered by step 110 and finished at 0.972 with more efficient solutions (turns 14.3 -> 11.8). Critic explained variance rose from a cold start (0.01 -> 0.07+) — no warm-start needed, unlike the compacted-PPO arm.
ppo — per-token terminal-reward stream, trainer-side
GAE (gamma=1.0, lambda=0.95), clipped surrogate (eps=0.2), clipped value
loss (coef 0.5), entropy_coef=0 (a 0.01 arm is a follow-up), LR 1e-6model.safetensors — step-150 policy, value head stripped (vLLM-loadable)value_head.safetensors — the trained PPO value head (value_head.weight,
[1, hidden]) for critic warm-start experiments
(trainer.model.ppo_value_head_init)<tool_call> JSON blocks).