paper-zero-rl-format-reward-4b

paper-zero-rl-format-reward-4b

Recipe: recipes/papers/zero-rl-format-reward · Collection: Papers, replicated

Zero RL on a base model: GRPO from Qwen3.5-4B-Base on MATH levels 3 to 5, strict boxed reward against correctness only. A rigid format reward costs accuracy. Both arms are read leniently at eval so the target is one number.

Result

Run 2026-09-18, both arms, one H100, 46.2 GPU minutes.

Recipe vs baseline: +0.094 [+0.052, +0.139] over 160 paired tasks. Both arms beat the base; the gap between them is the paper's finding and it clears the noise band the three base re-runs set. On the baseline's own strict reward the recipe arm scores lower (0.53 vs 0.58): it boxes less and is right more.

The six seed folders are the 2026-09-21 follow-up the README's Learned section asks for: three training seeds per arm at the same settings. Their paired read-out is not in the recipe README yet; the adapters are here so the numbers can be recomputed.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B-Base")
model = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b")  # the headline arm
model = PeftModel.from_pretrained(base, "while-ai/paper-zero-rl-format-reward-4b", subfolder="baseline")  # another arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/zero-rl-format-reward
python recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.