msr-agentic-ptb-opus-max

msr-agentic-ptb-opus-max — cell 3 backup

Backup of cell 3 of the agentic-PTB benchmark: Claude Code (claude-opus-5[1m]) at max reasoning effort, given Qwen/Qwen3.5-9B-Base, 4 GPUs and 100 autonomous hours to post-train it for agentic SWE / terminal work. The run completed its full window on 2026-08-17 18:28:54Z.

⚠ The submission is checkpoints/sft_v5_step_900

Not SUBMITTED_soup_base_sft. That directory name is a leftover from a six-hour window during the run and is wrong. The interpolated checkpoint it holds was submitted, then withdrawn by the arm on its own pre-registered decision rule ~6 h later, and sft_v5_step_900 was restored as the submission. checkpoints/soup_base_sft is the same weights under a non-misleading name. See SUBMITTED_soup_base_sft/WITHDRAWN.md.

Serve the submission directly — it is standard HF format:

vllm serve checkpoints/sft_v5_step_900 --served-model-name policy \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image":0,"video":0}' --max-model-len 65536

The submitted system is the checkpoint plus a harness plugin (pi-ws: a task working-directory fix and an appended operating-procedure prompt). The plugin is not in this repo — it lives at agentptb/runs/a-opus-max/workspace/pkgs/pi_ws. Without it the numbers below do not reproduce; the plugin is worth more than the weights (see below).

Results

swe-bench-verified on a seeded 250-task sample (SEED=0 pinned, so every run draws the identical sample), terminal-bench-2 on all 89 tasks. Every panel is a completed run with zero ungraded episodes, and every swe-bench cell is replicated.

Read the 2×2 before quoting any single number. All cells n=250, identical task set:

Two things follow, and the arm states both plainly in records/SUBMISSION.md:

1. The harness fix, not the training, carries the result. A working-directory fix plus a short prompt is worth +23pp on the untrained base.
2. The fine-tune is a net liability on the harness it submits. With the scaffold held fixed the weights are −6.8pp; the untrained base + scaffold (27.6%) beats the submitted system (20.2%). On the stock harness the same weights are +8.8pp. Both statements describe this checkpoint.

Do not report a terminal-bench-2 number for this cell. All 89 tasks, every comparison p≥0.125, and the base (5/89) is above the submitted system on both harnesses.

Noise floor, measured not assumed: two runs of an identical system disagree on 39 of 250 tasks (15.6%), symmetrically (21 vs 18). Any result near 22-vs-30 discordant is inside it.

Known packaging wart

config.json says "dtype": "float32" while the stored tensors are bf16. vLLM ignores the key and infers from the safetensors — every measurement above was logged at bf16. A transformers user loading the directory will upcast to fp32: numerically exact, twice the memory. Left as-is deliberately, because the measured artifact is the directory as it stands.

generation_config.json carries eos_token_id = [248044, 248046]. This matters: the base ships no generation config, so <|im_end|> — the token its own chat template ends every turn with — is not a stop token, and the model runs past its turn and hallucinates the user's reply.

What is here

• checkpoints/ — 14 checkpoints, every weight artifact the run produced and that still exists. Four of these were already deleted locally by prime-rl's keep_last rotation and survive only here. sft_v1 is absent because a node eviction at h6.75 destroyed it at step 600/940.
• records/ — SUBMISSION.md (the arm's own final write-up, 488 lines), NOTES.md (1,064 lines, 35 numbered findings), CHECKPOINT-CHAIN.md, DEADLINE.
• traces/ — the raw evaluation traces every number above was computed from. Every figure in this README has been independently recomputed from these by the operator and reproduces records/SUBMISSION.md exactly.
• data/ — the SFT corpora, including the three rewrite passes that produced sft_v5.

Full cross-cell comparison keys: agentptb/ROLLUP.md. Write-up: https://claude.ai/code/artifact/a965ff02-e109-49b4-b37c-5d4e47e8671f.

Caveat for cross-cell comparison

This cell ran under goal v5 on 3 GPUs (allocated 4, used 3). Cell 0 — the same driver at high effort — ran under goal v2 on 4. Goal v5 permits distillation and this arm also fixed the base's generation config. A cell-0 vs cell-3 difference therefore cannot be attributed to reasoning effort. A clean effort comparison needs an opus@high cell under goal v5.