grpo-vs-ppo-x16-candidates

Extensive evals for GRPO vs PPO (Qwen2.5-3B, DeepScaleR easy10k): dedup500 x16 selection + downstream suite

Status 2026-09-16. Arms: GRPO (g16), PPO priv0, PPO priv1 at 16 / 64 / 256 / 1024 / 4096 trajectories per update, seeds 42-44 (runs still training when this was frozen: PPO priv1 4096 s43 and s44, remote).

Plan

1. Stage 1 (in-distribution selection). For every finished run, every 20-step checkpoint <= 400 whose dedup500 x4 score is within 2.0 pp of that run's best x4 is re-scored on DeepScaleR dedup500 x16 (500 problems x 16 samples = 8,000 rows). Steps that already had x16 from the August GRPO sweep are excluded. Candidate list: models.tsv (225 rows: task, arm, key, traj, seed, step, source uri). Each run's highest x16 checkpoint is its representative; only that number is reported for the validation set.
2. Stage 2 (downstream). Each run's stage-1 winner is evaluated on MATH-500 x16 (8,000 rows), GSM8K x8 (10,552), AMC23 x32 (1,280) and MinervaMath x8 (2,176). Tables then read the same checkpoint for every benchmark.

Sampling for every eval: temperature 1.0, top_p 1.0, top_k -1, max 4,096 generated tokens, prompt <= 1,024 tokens, one sample per duplicated row (the x-factor is baked into the parquet), environment.env_class=deepscaler symbolic grader, trainer.seed=42.

Where everything is

• HF repo (public): JWei05/grpo-vs-ppo-x16-candidates
  • <task>/ policy weights (+ tokenizer/config) for every stage-1 candidate not yet evaluated plus the current x16 leader of every run (so stage 2 can start from HF alone): 158 folders. The first 29 uploaded are bf16 (~6.8 GB), the rest are the original fp32 exports (~13.6 GB) because converting on the loaded devbox was too slow; vLLM loads either (dtype=auto). Task names follow models.tsv.
  • eval_data/deepscaler_dedup500_x16.parquet (stage 1), eval_data/stage2_math500x16_gsm8kx8.parquet + eval_data/stage2_amc23x32_minervax8.parquet (stage 2; pass both).
  • results/x16_scores.csv every candidate with its x16 score where done (82/225 at freeze), results/models.tsv, results/hf_upload_list.tsv.
• Git (this folder): models.tsv, x16_scores.csv, models_remaining_hf.tsv (the 143 remaining candidates with hf://JWei05/grpo-vs-ppo-x16-candidates/<task> uris), results/<task>/aggregated_results.jsonl for finished tasks, build_manifest.py, select_best_x16.py, the two queue runners and run_x16_then_stage2.sh.
• Original fp32 exports stay on S3 (s3://scale-ml/genai/skyrl_dev/grpo_vs_ppo/runs/<run>/exports/0/global_step_N/policy/) and, for the 256-arm continuations and the remote PPO priv0 4096 s44 run, on HF under JWei05/<run>.

How to run the evals (remote box with GPUs)

The queue runner takes a TSV of models, a parquet (or several, comma-separated), and a list of GPUs; each GPU is one slot with its own Ray head + vLLM engine and pulls the next model as soon as it finishes. Model uris may be hf://<owner>/<repo>/<subdir> or s3://.../.

# 0. environment: SkyRL venv from this repo (or the scale-train image); HF_TOKEN not needed (public repo)
hf download JWei05/grpo-vs-ppo-x16-candidates --include "eval_data/*" "results/*" --local-dir ./x16   # parquets + manifests

# 1. stage 1: dedup500 x16 on the remaining candidates
MODELS_TSV=docs-ppo/plots/dedup500_x16/models_remaining_hf.tsv \
EVAL_DATA=./x16/eval_data/deepscaler_dedup500_x16.parquet RESULTS=docs-ppo/plots/dedup500_x16/results \
GPUS="0 1 2 3" bash docs-ppo/plots/dedup500_x16/run_local_eval_queue.sh          # or run_local_eval_queue_docker.sh (needs a SkyRL docker image; DATA_ROOT=./x16/eval_data)

# 2. pick each run's best x16 -> stage2_models.tsv (also prints the study-wide max)
python docs-ppo/plots/dedup500_x16/select_best_x16.py docs-ppo/plots/dedup500_x16/results docs-ppo/plots/dedup500_x16/models.tsv docs-ppo/plots/dedup500_x16/stage2_models.tsv
#    (models.tsv rows carry S3 uris; for an HF-only box rewrite column 7 to hf://JWei05/grpo-vs-ppo-x16-candidates/<task>)

# 3. stage 2 on the winners
MODELS_TSV=docs-ppo/plots/dedup500_x16/stage2_models.tsv \
EVAL_DATA=./x16/eval_data/stage2_math500x16_gsm8kx8.parquet,./x16/eval_data/stage2_amc23x32_minervax8.parquet \
RESULTS=docs-ppo/plots/dedup500_x16/results_stage2 GPUS="0 1 2 3" bash docs-ppo/plots/dedup500_x16/run_local_eval_queue.sh

Scores land in RESULTS/<task>/aggregated_results.jsonl as eval/<data_source>/avg_score (data sources: deepscaler_dedup500, math500, gsm8k, amc23, minervamath). Runners are restart-safe (a task with _SUCCESS is skipped); one 8,000-row task takes ~9 min on a free H100 (fp32 export) and the whole remaining stage 1 is ~5.5 GPU-hours per 4 GPUs, stage 2 ~6 GPU-hours. The single main_eval command a slot runs is spelled out inside the runner if you prefer your own scheduler.

Stage 1 progress at freeze (82 / 225 candidates scored)

Why frozen here: the local devbox GPUs were shared with another user's jobs (see TRACKING Run 049); the remaining work moves to a remote box. Scores so far confirm the x4-vs-x16 gap is small (x16 typically 0.5-1.5 pp below the x4 best at the same step).