Downloads · 30 days
6
13% of all-time downloads
lyx02klmy/qwen3-4b-cp26-grpo-1step-bsz128
qwen3-4b-cp26-grpo-1step-bsz128 is a text generation model from lyx02klmy. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A Qwen/Qwen3-4B checkpoint fine-tuned with one GRPO step on the circle-packing problem (pack 26 unit-radius-summed circles into a unit square). The single update was deliberately large: 1024 rollouts in one batch (128…
Downloads · 30 days
6
13% of all-time downloads
All-time downloads
48
Public
Parameters
4.4B
8.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8.8 GB · 100%
From the Hugging Face model README
A Qwen/Qwen3-4B checkpoint fine-tuned with one GRPO step on the circle-packing problem
(pack 26 unit-radius-summed circles into a unit square). The single update was deliberately large:
1024 rollouts in one batch (128 prompts × 8 rollouts each), no entropy bonus, no KL anchor.
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen3-4B (4.02B params) |
| Trainer | verl PPO loop |
| Algorithm | GRPO (adv_estimator=grpo, group-relative advantage) |
| Train batch size | 128 prompts (full dataset) |
| Rollouts per prompt | n = 8 |
| Total rollouts in the step | 1024 |
| Total training steps | 1 |
| Optimizer | AdamW, lr = 5e-6 |
kl_loss_coef | 0.0 (no KL anchor to base) |
entropy_coeff | 0.0 |
| Sampling temperature | 1.0 |
| Max response length | 32,768 tokens |
| GPUs | 2 × A100-80GB, FSDP, vLLM rollout backend |
ppo_mini_batch_size / ppo_epochs | 128 / 1 |
| Reward | constrained-optimization shape: U = sum_r if valid; U = max(sum_r − λ·violation, −1) if near-valid with sum_r ≤ 2.64; U = −1 otherwise. λ = 1.0, anti-gaming cap at 2.64 |
Each prompt asks the model to write a Python run_packing() function that returns
(centers, radii, sum_radii) for n=26 non-overlapping circles inside the unit square.
Reward = sum-of-radii of a valid packing, with a penalty term for near-valid responses
that violate overlap or out-of-bounds constraints. Known optimum for n=26 ≈ 2.636.
ftol=1e-12, maxiter=1e6).use_kl_loss=False, kl_loss_coef=0) and no entropy bonus
(entropy_coeff=0); these were intentionally disabled to study GRPO's group-relative signal
in isolation. With a 1-step budget, we don't see the mode collapse that would emerge over many steps.step_1_responses.jsonl was overwritten by val_on_last_step after the
step, so the per-rollout response log from training is not preserved.The model emits <think>...</think> chain-of-thought followed by a ```python code block
containing run_packing(). To extract the code: strip <think> content, then grab the first
fenced code block. See verl/utils/reward_score/circle_packing.py in the verl repo for the
exact extraction + scoring used during training.
If you use this checkpoint, please cite the underlying verl framework and the circle-packing benchmark setup.