qwen-1.7b-math-reasoner-grpo

Qwen3-1.7B-GRPO-GSM8K (v1)

A GRPO (Group Relative Policy Optimization) fine-tune of unsloth/Qwen3-1.7B-Base, trained with verifiable rewards (RLVR) on GSM8K math word problems for one full epoch.

Model description

• Base model: unsloth/Qwen3-1.7B-Base (Apache 2.0)
• Method: GRPO (DeepSeekMath / DeepSeek-R1 style), via Unsloth + TRL's GRPOTrainer
• Adapter: LoRA rank 32 / alpha 32, merged into the base weights for this release
• Training data: GSM8K train+test combined into one 8,792-example pool, shuffled once (seed 3407), split 80/10/10 — trained on the 80% partition (7,033 examples) for 1 full epoch. Validation (10%) used only for monitoring during training; the eval partition (10%, 880 examples) was never touched until the final evaluation below.
• Output format: the model is trained to respond in a fixed structure:

  <reasoning>
  ...
  </reasoning>
  <answer>
  ...
  </answer>
  

• Reward functions: correctness (regex-extracted <answer> content vs. GSM8K ground truth, +2.0), format compliance (+0.5 well-formed / +0.25 partial), brevity (+0.2 max, capped small so it can never outweigh correctness or format)

Evaluation results

All numbers below are from held-out data the model never saw during training or any prior evaluation pass (see the source repo's week-05-06/scripts/data_split.py for the split methodology, and week-05-06/docs/04-results.md for the full writeup including two scoring bugs that were caught and fixed mid-evaluation).

*Base model accuracy is scored leniently (correct number anywhere in the response) since the base model has no tendency to use the <answer> tag format at all (0.6% format compliance) — this model's own format compliance is 99.4%.

**MATH's numbers are a lower bound, not a precise score — grading MATH answers exactly (fractions, intervals, symbolic expressions) without a computer algebra system is a known hard problem; some correct-but-differently-formatted answers score as incorrect here.

Manual transcript review (not just the aggregate numbers) found no evidence of reward hacking: correct answers show genuine multi-step arithmetic, incorrect answers are real reasoning mistakes on harder problems (not degenerate or gamed output), and response length does not correlate with correctness in a gaming direction.

Independent cross-check: lm-evaluation-harness (base vs. trained)

The numbers above use this project's own zero-shot, custom-prompt eval scripts. Cross- checked independently via EleutherAI's lm-evaluation-harness (standard 5-shot GSM8K config, plus MMLU and HellaSwag as general-capability anchors), run identically against both the base model and this one — see the source repo's week-07/docs/04-results.md for the full writeup.

No catastrophic forgetting. The MMLU and HellaSwag deltas are smaller than each benchmark's own standard error (~0.4-0.5pp) — statistically indistinguishable from no change. GSM8K improved well outside that noise band, via a completely different methodology (5-shot, standard harness prompt format) than this project's own eval — independent confirmation that the math-reasoning gain is real, not an artifact of this project's own eval design, and that it wasn't purchased at the cost of general capability.

Limitations

• Narrow, math-specialized training. This model was trained exclusively on grade- school arithmetic word problems (GSM8K). General instruction-following and chat quality specifically were not evaluated (this model was never trained on conversational data) — though catastrophic forgetting of general knowledge/reasoning capability was checked (see the benchmark cross-check above) and not found.
• A real hint-exploitation vulnerability was found, quantified at 37%. A formal hint/no-hint probe (200 held-out problems, each run with a correct hint and a deliberately wrong decoy — see week-07/docs/04-results.md for the full method) found that 37% of the time, a plausible-but-wrong hint changes the model's final answer away from what it would otherwise get right. In one documented example, the model correctly re-derived the right answer three separate times, explicitly noted the conflict with the hint, and still deferred to the hint over its own confirmed work. Do not present this model with untrusted "hint" or "suggested answer" text in the prompt without accounting for this - it will sometimes follow a wrong suggestion over independently- verified correct reasoning. (Checked and ruled out as reward hacking specifically - training prompts never included hints at all, so the reward function never selected for this behavior one way or the other; it's most likely inherited from base-model pretraining and was simply never tested or corrected during RL training.)
• MATH evaluation numbers are approximate (see above) — treat the MATH result directionally, not as a precise benchmark score.
• Single training run. No hyperparameter ablation or multi-seed variance check was performed; this is a v1 release, not a fully tuned final model.
• Format compliance is 99.4%, not 100% — a small fraction of outputs don't follow the <reasoning>/<answer> structure.
• The model does not always stop generating after </answer> — measured at ~0.07% of outputs, rare rather than common. No reward function ever scored anything past the first <answer> tag match, so nothing in training penalized rambling afterward — a real design gap. Measured directly against 1,451 raw eval transcripts (GSM8K + SVAMP + MATH), only 1 case had any content past the tag, and it wasn't even the fabricated-dialogue pattern (it was a repetitive reasoning loop that leaked one stray character). The one case that clearly showed the fabricated-dialogue version happened during manual testing on a far-out-of-distribution question style — possibly the same underlying cause as the function-composition gap below, not a frequent, independent problem. Regardless, the fix is free and worth applying — set a stop sequence at </answer> when generating:

  # vLLM
  SamplingParams(stop=["</answer>"], include_stop_str_in_output=True)
  
  # transformers .generate()
  from transformers import StopStringCriteria, StoppingCriteriaList
  model.generate(..., stopping_criteria=StoppingCriteriaList(
      [StopStringCriteria(tokenizer, ["</answer>"])]
  ))
  

• Symbolic function-composition problems (f(x)=..., g(x)=... notation) appear to be a specific weak spot, distinct from the general MATH miss rate — likely because GSM8K (the only training data) is exclusively concrete word problems with no examples of this abstract notation. Observed anecdotally (4/4 wrong on manual testing), not verified at benchmark scale — flagged as a real, specific gap rather than confirmed precisely.

Training details

• Framework: Unsloth 2026.5.9 + TRL GRPOTrainer, run inside the unsloth/unsloth Docker image on an RTX 5070 (12GB, Blackwell sm_120)
• Precision: bf16 (FP8 was evaluated and found broken on this specific software stack — see the source repo's week-05-06/docs/00-overview.md for the investigation)
• num_generations=4, temperature=0.8, beta=0.001 (KL penalty), loss_type=dapo
• Learning rate 5e-6, 1 epoch, ~7,033 steps, wall-clock ~7.4 hours

Citation / provenance

Trained as part of a self-directed practice roadmap covering post-training/RL, systems, and scalability topics for LLMs.