msr-agentic-ptb-opus-high

msr-agentic-ptb-opus-high — cell 0 backup

Backup of cell 0 of the agentic-PTB benchmark: Claude Code (claude-opus-5[1m]) at high reasoning effort (the CLI default), given Qwen/Qwen3.5-9B-Base, 4 GPUs and 100 autonomous hours to post-train it for agentic SWE / terminal work. The run completed its full window (2026-08-09 09:59:29Z → 2026-08-13 13:59:29Z).

Submission: checkpoints/step20_swe8pct — GRPO step 20, selected on a held-out SWE-smith slice.

⚠ Ignore the _swe8pct in that directory name

It records an early reading that the arm itself retracted. The name was fixed at upload time and cannot be changed without breaking references, so treat it as an opaque label.

Results — the cell finished by overturning its own headline

An earlier headline of +3.50pp on 200 tasks did not survive the full suite. The cause is worth knowing before you sample this benchmark yourself: the leading tasks of SWE-bench Verified are alphabetically ordered, not a random sample. Watching the paired estimate accumulate as tasks were added:

+3.90 (n=77)  →  −0.60 (n=168)  →  −2.87 (n=279)  →  −0.60 (n=498)

The honest summary is the arm's own: "the RL checkpoint is directionally better on both suites and neither difference is statistically resolved... consistent with a small improvement, not evidence of one." The most valuable output of this cell is the measurement discipline, not the weights.

What is here

• checkpoints/ — step20_swe8pct (the submission), step60, step80. step_40 is absent: prime-rl's keep_last=3 rotation deleted it before it could be copied, and it is unrecoverable. Selection used a held-out SWE-smith slice (images 9–24), never the reported suites, and it chose the earliest checkpoint because 80 steps did not beat 20.
• records/ — SUBMISSION.md, NOTES.md, RESULTS.md, RESULTS_raw.md, HOLDOUT.md.

Full cross-cell comparison keys: agentptb/ROLLUP.md. Write-up: https://claude.ai/code/artifact/95e75b6b-a5cd-4172-b165-c9f4d0673b25.

Caveat for cross-cell comparison

This cell ran under goal v2 on 4 GPUs and did no SFT — GRPO directly from base. Cell 3 (msr-agentic-ptb-opus-max, the same driver at max effort) ran under goal v5 on 3 GPUs and did SFT only. Goal v5 permits distillation, and cell 3 also fixed the base's missing generation_config.json. A cell-0 vs cell-3 difference therefore cannot be attributed to reasoning effort. A clean effort comparison needs an opus@high cell under goal v5.

Note also that base rates in this project are protocol-dependent — the same untrained base reads 5.00% at max_concurrent=32 and 9.10% under the final avg@2 protocol. Only compare within an identical harness and protocol.