qwen-grpo-r5

Qwen3-0.6B + GRPO on GSM8K

GRPO (TRL GRPOTrainer) applied to Qwen3-0.6B, the smallest model in the Qwen3 family, with a verifiable correctness reward on GSM8K.

+23.4 points absolute / +49% relative over the untrained baseline, for $7.96 of GPU time.

Important: this model runs in non-thinking mode

Its chat template is patched so the generation prompt always ends with an empty <think>\n\n</think> block. Thinking mode is not available — the 0.6B model cannot finish a reasoning block inside a usable token budget, and training/eval formats are kept identical on purpose. Use the tokenizer that ships with this repo.

from transformers import AutoModelForCausalLM, AutoTokenizer

m = AutoModelForCausalLM.from_pretrained("cmpatino/qwen-grpo-r5", dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained("cmpatino/qwen-grpo-r5")

PROMPT = """Solve the following math problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.

{q}

Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \\boxed command.

Reasoning:"""

msgs = [{"role": "user", "content": PROMPT.format(q="Natalia sold clips to 48 friends in April, and then she sold half as many clips in May. How many clips did she sell altogether?")}]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(m.device)
print(tok.decode(m.generate(ids, max_new_tokens=512, do_sample=False)[0][ids.shape[-1]:]))

Reward

The reward is a line-by-line reimplementation of the scorer the benchmark actually uses — inspect_ai's match(numeric=True, location="end"): strip $ , £ € * _ and trailing periods, split the completion on whitespace, scan tokens in reverse, and compare the first parseable number against the gold answer at 5 significant figures. It was unit-tested against the real scorer before training. A second reward (weight 0.2) pays for ending on an ANSWER: <number> line; it saturates above 0.95 within ~25 steps.

Training

Three sequential GRPO stages on openai/gsm8k main train (zero-shot prompts, inspect's MATH_PROMPT_TEMPLATE verbatim), 251 optimizer steps total — about 0.55 of one epoch, so no prompt is seen twice.

DAPO loss, beta=0 (no KL penalty, no reference model), rewards scaled within each rollout group, truncated completions masked out, max_completion_length 768, vLLM colocated with the trainer on a single GPU.

Intermediate checkpoints are published as cmpatino/qwen-grpo-r4-s{25,50,75,100,125} and cmpatino/qwen-grpo-r5-s60. Full-test scores rise roughly monotonically with cumulative steps (0.646 → 0.662 → 0.658 → 0.683 → 0.692 → 0.708) and then flatten: qwen-grpo-r4-s125 scores 0.7081 ± 0.0125, a statistical tie with this model.

Caveats

• The reported score is r5's final checkpoint, so it involves no test-set selection — but r5 was started from r4-s100, picked using limit-200 test scores. Read 0.708–0.709 as one plateau, not two results.
• No validation split was held out of GSM8K train; the eval budget went into making the reported numbers full-test instead.
• Standard error is ±0.013, so checkpoint differences below ~0.03 are noise.
• Greedy decoding, single sample, canonical 10-shot inspect_evals/gsm8k. No self-consistency.
• Optimising directly against the benchmark's own scorer is deliberate here; it means the score should be read as "GSM8K-shaped arithmetic reasoning", not as general math ability.