Qwen3-4B-Base-MSJ-GRPO-DAPO17K

Qwen3-4B-Base-MSJ-GRPO-DAPO17K

Final actor checkpoint from a completed Qwen3-4B-Base GRPO run on DAPO Math 17K with an auxiliary mistake-conditioned SFT objective. The repository root contains the merged Hugging Face model for evaluation and inference.

Checkpoint

The root model-*.safetensors files contain the merged actor weights from training step 204 of 204. They load directly with Transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SriGoli0123/Qwen3-4B-Base-MSJ-GRPO-DAPO17K"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

The merged export contains 4,022,468,096 bfloat16 parameters across 398 tensors. The final deterministic DAPO Math validation accuracy was 0.325 on 1,000 examples at step 204 (best during training: 0.348 at step 200).

Training setup

• Base model: Qwen/Qwen3-4B-Base
• Base revision: 906bfd4b4dc7f14ee4320094d8b41684abff8539
• Dataset: DAPO Math 17K
• Training examples: 13,116
• Validation examples: 1,000
• Framework: VeRL (SriGoli-0123/verl-a100, branch a100/ioher-mistake-judge, commit d169b69)
• Hardware: 4 NVIDIA A100 80GB GPUs (actor and SGLang rollout colocated)
• Objective: GRPO plus mistake-conditioned auxiliary SFT
• Epochs: 1
• Completed training steps: 204
• Exported checkpoint: step 204
• Validation frequency: every 5 steps
• Checkpoint frequency: every 10 steps and the final step
• Validation before training: disabled

Batching and sequence lengths

• Batch size (data.train_batch_size): 64 prompts
• Rollouts per prompt (rollout.n): 8
• Trajectories per step: 512
• Mini-batch size (actor.ppo_mini_batch_size): 16 prompts (4 optimizer steps per training step)
• Micro-batch size per GPU (actor.ppo_micro_batch_size_per_gpu): 4
• Log-prob micro-batch size per GPU (rollout.log_prob_micro_batch_size_per_gpu): 4
• Dynamic batch size (actor.use_dynamic_bsz, rollout.log_prob_use_dynamic_bsz): enabled
• Max tokens per GPU (actor.ppo_max_token_len_per_gpu, rollout.log_prob_max_token_len_per_gpu): 24,576
• PPO epochs (actor.ppo_epochs): 1
• Max prompt length (data.max_prompt_length): 2,048 tokens
• Max response length (data.max_response_length): 4,096 tokens
• Judge max prompt tokens (algorithm.ioh.judge_max_prompt_tokens): 8,192
• Judge max new tokens (algorithm.ioh.judge_max_new_tokens): 512
• Max extra prompt tokens for inoculation (algorithm.ioh.max_extra_prompt_tokens): 256

GRPO configuration

• Advantage estimator: GRPO
• Policy loss: vanilla
• Loss aggregation: token mean
• Discount factor: 1.0
• Lower clipping ratio: 0.20
• Upper clipping ratio: 0.28
• Dual-clip coefficient: 3.0
• Entropy coefficient: 0
• Gradient clipping: 1.0
• KL loss coefficient: 0
• KL contribution to reward: disabled

Auxiliary SFT and judge

• SFT coefficient: 0.01
• Combined objective: GRPO loss + 0.01 * SFT loss
• SFT loss aggregation: token mean
• GRPO and SFT gradients are applied in the same optimizer step
• SFT rows packed without filler rows (same loss as unpacked)
• Judge input: question, negative rollout, ground truth, and 3 few-shot examples (all kept for every request)
• Judge model: current policy model
• Judge requests overlapped with the rollout tail
• Feedback extraction: body of the first complete <mistakes>...</mistakes> block
• Negative-rollout reward threshold: 0.5
• Inoculation injection location: system message
• Invalid judge feedback: excluded from SFT

Optimization

• Optimizer: PyTorch AdamW
• Learning rate: 1e-6
• Weight decay: 0.01
• Adam betas: [0.9, 0.999]
• Learning-rate scheduler: constant
• Warm-up steps: 15

Distributed and rollout configuration

• Actor strategy: four-way FSDP1
• Model precision: bfloat16
• Parameter offloading: disabled
• Optimizer offloading: disabled
• Gradient checkpointing: enabled
• torch.compile: disabled
• Padding removal: enabled
• Attention: FlashAttention-2
• Rollout engine: SGLang 0.5.8
• Tensor parallelism: 1
• Rollout temperature: 1.0
• Top-p: 1.0
• Top-k: disabled (-1)
• SGLang memory-utilization target: 0.60

Final-step audit

• Training rollouts: 512
• Correct rollouts: 82
• Negative rollouts: 430
• Judge candidates: 430
• Judge feedback accepted into SFT: 389
• Judge feedback rejected as malformed: 41
• Rollouts skipped before judging: 0

Limitations

This is a research checkpoint. Validation accuracy plateaued around 0.31-0.33 after roughly step 70; differences of about ±1.5 points are within noise on the 1,000-example validation set. Evaluate the model on the intended benchmark and decoding configuration before downstream use.