Qwen3-8B-GRPO-MATH

Qwen3-8B-GRPO-MATH

Qwen3-8B trained with GRPO on the Hendrycks MATH training split, using verifiable final-answer rewards. Produced as the RL teacher for a study on how much of a teacher's RL gain survives on-policy distillation into a smaller student.

This is a short-budget research checkpoint, not a tuned release. It is 120 optimizer updates over roughly a quarter of one epoch. See Honest limitations below before using it.

Training

Prompt coverage is 16 x 120 = 1,920 of 7,496 prompts, i.e. 0.26 epoch.

Reward-function caveat worth repeating

slime ships three math scorers and only one is correct for these labels. Verified on 300 real MATH labels by constructing responses that are correct by definition:

deepscaler splits on </think> and returns 0 for every non-thinking response; dapo needs a literal Answer: line. Only ~64% of MATH labels are numeric, the rest are symbolic LaTeX, and math handles all 94 symbolic labels in the sample. This model was trained with --rm-type math.

Evaluation

avg@16 / pass@16, non-thinking chat mode, temperature 1.0, top-p 1.0, no top-k, scored with the same harness for every row.

AIME 2024 + 2025, at a 16,384-token budget where truncation is 0-1% and the headline number is no longer budget-limited:

MATH-500, at a 4,096-token budget:

pass@16 rises alongside avg@16 on every benchmark, so this run expanded the set of solvable problems rather than only sharpening the top-1 answer.

Read the budget before you read the score

The model roughly doubles its response length (2,325 -> 5,203 tokens), and a truncated response scores exactly zero. The same checkpoint measured at smaller budgets:

At 4,096 tokens you are largely measuring how often it finishes, not whether it can solve the problem. Give it 16k.

Honest limitations

• Short budget. 120 updates, 0.26 epoch of the 7,496 prompts. Not a converged RL run.
• Weak gradient signal. MATH is easy for Qwen3-8B: mean rollout reward ran 0.65-0.97, and 12-13 of every 16 prompt groups had zero reward variance, so ~78% of rollout compute produced no gradient. A harder prompt set or DAPO-style dynamic sampling would help.
• Length inflation. See the budget table above. This is the single easiest way to mis-measure this model.
• The training response cap matters enormously. An otherwise identical run with the rollout cap at 2,048 instead of 4,096 tokens reached only +2.04 overall and lost a point on MATH-500. The cap, not the step count, was the binding constraint.
• Single seed. AIME24/25 are 30 problems each; repeated avg@16 evaluations of the same model differ by about +/-2 points. Treat any difference under 2 points as noise.

Intended use

Research on RL-for-reasoning and on-policy distillation. Not intended for deployment.

License

Apache-2.0, inherited from Qwen3-8B. The MATH dataset is MIT-licensed.