sqa-dqo-a1-step400

sqa-dqo-a1-step400

DQO (alpha=1.0) baseline on ScienceQA, trained with GRPO on Qwen/Qwen2.5-Math-1.5B.

Selected as best validation pass@6 for this arm (rank 1).

Diversity-Quality Optimization (Chen et al., ICLR 2026): adds alpha*logdet(L+I) to the objective with a leave-one-out estimator. alpha=1.0 is the paper's setting. NOTE: we use reference-policy hidden states as the embedding phi to match STRIDE, so the comparison isolates the diversity objective rather than the representation; the original implementation uses a pretrained sentence encoder.

Training data and format reward

The training prompt requests the numbered-step scaffold and a final \boxed{X}, matching the format reward. It does not carry the "IMPORTANT rules" emphasis block used by the AoPS suite, so the wording (not the requested output format) differs from that suite.

Arm-specific: entropy_coeff=0.0, clip=0.2/0.2, rollout temperature=1.0, DQO alpha=1.0 (logdet diversity term, ref-policy embedding).

Do not apply a chat template

Trained on raw prompt text. verl's RLHFDataset has apply_chat_template=False and it was never enabled. Applying Qwen2.5-Math's chat template at inference creates a train/eval mismatch measured at roughly 19 points of pass@1.

from vllm import LLM, SamplingParams
llm = LLM(model="sandeep123/sqa-dqo-a1-step400", dtype="bfloat16", max_model_len=1536)
params = SamplingParams(n=6, temperature=1.0, top_p=1.0, top_k=-1, max_tokens=1024)
out = llm.generate([prompt_text], sampling_params=params)   # raw string, not llm.chat()

Validation metrics at this checkpoint

Answer extraction (pre-registered). An answer is the content of the final \boxed{}; if absent, the last standalone A-E token. Responses with no extractable answer are scored incorrect, and all K rollouts stay in the denominator.

Validation uses 256 held-out prompts, K=6, temperature 1.0, seed 42 -- pinned in code so every arm, including the temperature-1.2 arm, is scored under identical decoding.

Checkpoint selection

Quality-optimal and diversity-optimal checkpoints differ substantially, so both the best-pass@1 and best-pass@6 checkpoints are published for every arm. Diversity results are therefore never reported from a checkpoint chosen purely for accuracy.