Qwen3.6-35B-A3B-caesar-cipher-GRPO-20260924

Qwen3.6-35B-A3B caesar_cipher GRPO teacher

Full-parameter GRPO of Qwen3.6-35B-A3B on the reasoning_gym caesar_cipher task, trained as a frozen teacher for multi-teacher on-policy distillation (MOPD). Thinking is on.

This revision: training step 125 (weights = policy version 125, the exact weights the step-125 eval measured). Every upload is tagged step-N.

Held-out eval

reasoning_gym caesar_cipher, default config, held-out problems (seed 10000, disjoint from training), 4 rollouts each at temperature 1.0, top_p 1.0, 32k context, thinking on. Reward is exact match (reasoning_gym score_answer == 1) on the <answer> after </think>.

Training

• Framework: prime-rl (full fine-tune, no LoRA), one node of 8 B200: 4 GPUs vLLM TP4, 4 GPUs trainer (FSDP, EP4).
• GRPO, AdamW lr 1e-6, weight decay 0, bf16 optimizer states.
• 32 prompts x 8 rollouts per step; groups with no reward variance are dropped and replaced (DAPO-style dynamic sampling), so each step trains on 256 rollouts that carry signal.
• 1-step async rollouts (max 2 steps off-policy); temperature 1.0; up to 30,720 completion tokens.
• Data: reasoning_gym 0.1.25 caesar_cipher, default config, seed 20000 train pool; prompts over 2,048 tokens dropped. System prompt: reasoning_gym SYSTEM_PROMPTS['default'].
• Metrics: W&B project opd-rl-teachers, run caesar_cipher-grpo-3x2eng-20260925.
• Last logged training step when this card was written: 130.