aops-grpo-temp12-step600

aops-grpo-temp12-step600

GRPO + temperature 1.2 baseline on AoPS, trained with GRPO on Qwen/Qwen2.5-Math-1.5B.

Selected as best validation pass@6 for this arm (rank 2).

Exploration via the sampling distribution on AoPS. Trained at rollout temperature 1.2 but VALIDATED at 1.0 like every other arm, so its curve stays comparable.

Training data and format reward

Full boxfix protocol: the prompt demands Step 1: ... Step n: followed by \boxed{X} on its own line with nothing after it, and the 0.03 format reward pays for exactly that structure.

Arm-specific: entropy_coeff=0.0, clip=0.2/0.2, rollout temperature=1.2.

Do not apply a chat template

Trained on raw prompt text. verl's RLHFDataset has apply_chat_template=False and it was never enabled. Applying Qwen2.5-Math's chat template at inference creates a train/eval mismatch measured at roughly 19 points of pass@1.

from vllm import LLM, SamplingParams
llm = LLM(model="sandeep123/aops-grpo-temp12-step600", dtype="bfloat16", max_model_len=3072)
params = SamplingParams(n=6, temperature=1.0, top_p=1.0, top_k=-1, max_tokens=2048)
out = llm.generate([prompt_text], sampling_params=params)   # raw string, not llm.chat()

Validation metrics at this checkpoint

Answer extraction (pre-registered). An answer is the content of the final \boxed{}, compared by mathematical equivalence. Responses with no extractable answer are scored incorrect, and all K rollouts stay in the denominator.

Validation uses 256 held-out prompts, K=6, temperature 1.0, seed 42 -- pinned in code so every arm, including the temperature-1.2 arm, is scored under identical decoding.

Checkpoint selection

Quality-optimal and diversity-optimal checkpoints differ substantially, so both the best-pass@1 and best-pass@6 checkpoints are published for every arm. Diversity results are therefore never reported from a checkpoint chosen purely for accuracy.