Downloads · 30 days
0
0% of all-time downloads
just1nseo/qwen3-4b-ifrlvr-grpo-judge-30ba3b
qwen3-4b-ifrlvr-grpo-judge-30ba3b is a text generation model from just1nseo. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Qwen3-4B trained with GRPO on instruction-following data, using a verifiable IFEval constraint reward plus a bonus from an LLM judge (Qwen3-30B-A3B). Trained with verl. Thinking mode is retained — the policy is trained
Downloads · 30 days
0
0% of all-time downloads
All-time downloads
62
Public
Repo size
24.1 GB
Likes
0
Public
Click a slice to open those files.
.safetensors24.1 GB · 100%
From the Hugging Face model README
Qwen3-4B trained with GRPO on instruction-following
data, using a verifiable IFEval constraint reward plus a bonus from an LLM judge
(Qwen3-30B-A3B). Trained with
verl. Thinking mode is retained — the policy is trained
with <think> reasoning enabled.
Every checkpoint lives in its own global_step_*/ folder, one per epoch. Load one with
subfolder=.
| folder | steps | epoch |
|---|---|---|
global_step_91/ | 91 | 1 |
global_step_182/ | 182 | 2 |
global_step_273/ | 273 | 3 (final) |
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "just1nseo/qwen3-4b-ifrlvr-grpo-judge-30ba3b"
subfolder = "global_step_273" # final; or global_step_91 / global_step_182
tok = AutoTokenizer.from_pretrained(model_id, subfolder=subfolder)
model = AutoModelForCausalLM.from_pretrained(
model_id, subfolder=subfolder, dtype="bfloat16", device_map="auto"
)
msgs = [{"role": "user", "content": "Write exactly 3 bullet points about the ocean, all in lowercase."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=1024)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Each rollout scores
reward = ifeval_constraint_accuracy + 0.1 * [judge_score >= 5.3]
[0, 1]. Responses that never close their <think> block (i.e. ran
out of generation budget mid-reasoning) score 0.0.+0.1 is paid at or above the threshold. Because the judge emits
integers only, the 5.3 threshold is equivalent to >= 6. The judge is served with the Qwen3
non-thinking sampling preset (temperature=0.7, top_p=0.8, top_k=20, min_p=0).Maximum achievable reward per rollout is therefore 1.1. No perplexity/anchor shaping term is used in this run.
| Base model | Qwen/Qwen3-4B |
| Algorithm | GRPO (verl), 8 rollouts per prompt |
| Epochs / steps | 3 / 273 (91 steps per epoch) |
| Train batch | 1024 prompts, mini-batch 1024 |
| LR | 1e-6 |
| KL | use_kl_loss=True, kl_loss_coef=0.001, not applied in reward |
| Entropy coeff | 0 |
| Max prompt / response | 2048 / 8192 tokens |
| Precision | bf16 (FSDP), vLLM 0.11 rollout |
| Hardware | 8 × H100 80GB, 66.8 h wall clock |
The judge ran co-resident with the policy on the same 8 GPUs (4 × TP=2 replicas), using vLLM sleep/wake to hand the GPUs back to the trainer between rollout and update phases.
Measured on the training batches (not held out):
| step | total reward | constraint acc | judge score | judge pass rate | response len | entropy |
|---|---|---|---|---|---|---|
| 1 | 0.544 | 0.507 | 5.38 | 0.484 | 2621 | 0.400 |
| 91 | 0.721 | 0.672 | 5.77 | 0.548 | 1673 | 0.377 |
| 182 | 0.806 | 0.747 | 6.21 | 0.626 | 1895 | 0.428 |
| 273 | 0.854 | 0.786 | 6.68 | 0.709 | 2042 | 0.453 |
Both reward components rise together, and mean response length dips through epoch 1 (2621 → 1673) before recovering to 2042 by the end — this run does not show the length collapse that a constraint-only reward produces on this base model.