r2egym-rad-negative-warmup

RAD negative-warmup on R2E-Gym — Qwen3.5-4B, sampled trajectory

One RAD negative-warmup run (nwb0921a) on the R2E-Gym subset, starting from Qwen/Qwen3.5-4B, published every five updates so the run can be studied as a trajectory rather than an endpoint, plus step50, the run's best checkpoint:

Use step50 unless you want a point on the trajectory: it scores highest of every checkpoint measured on SWE-bench Verified (see below). The run continued to update 60 and those checkpoints are deliberately not published, because the policy degrades badly past update 50.

stepN holds the policy after N committed updates, converted from Megatron dist-checkpoint iter_{N-1:07d}. The run saved a checkpoint at every update; the intermediate ones are retained on the training host and can be published on request. step20 is where the negative warmup reaches its full bound and step40 is the run's final policy.

Usage

Pass the subfolder explicitly — the repo root holds no weights:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ziansu/r2egym-rad-negative-warmup"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="step50", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="step50")

AutoModelForCausalLM yields Qwen3_5ForCausalLM (4.21 B parameters, the language model). The checkpoints also carry the base model's vision tower, so AutoModelForImageTextToText yields the full Qwen3_5ForConditionalGeneration (4.54 B). Agent evaluation in this project used the language model only.

Training

Held-out trajectory

Two held-out measurements exist, at different resolutions. The 71-task R2E holdout is the one to read: 213 episodes per point (71 tasks x seeds 42/43/44), the same frozen manifest and protocol used for the other runs in this project.

Every trained checkpoint is 7-11 points above the base model. Across the trained points the trajectory is flat: the binomial standard error at 213 episodes is about 3.4 points, and the four values span 3.8 points, so they are mutually within roughly one standard error. Update 40 is the highest of them, and its per-seed values are unusually spread (57.7 / 47.9 / 63.4) compared with about 3 points of spread elsewhere.

The run also logged a smaller 32-task evaluation during training, every ten updates: 0.531, 0.594, 0.594, 0.469 at updates 10/20/30/40. That last value looks like a collapse and is not one - one task is 3.1 points there and the standard error is about 8.8 points. The 213-episode holdout above, measured on the same checkpoints, puts update 40 at the top. Prefer the 71-task numbers.

At update 40 the mean token weight is negative (-0.0284) with 16.1 % of tokens pinned at the lower bound and 13.8 % at the upper. That is the negative-warmup schedule reaching its configured bound; on the evidence above it did not cost held-out performance by update 40.

Per-update training reward carries no trend information at all: each update scores 32 fresh task groups, whose binomial standard error (about 0.088) accounts for essentially all of the observed update-to-update spread (sd 0.087).

SWE-bench Verified

All 500 tasks, three grouped rollouts per task at seed 42 (1,500 attempts), 65,536-token context, 75 tool turns, temperature 1.0 - the same scaffold and sampling settings used in training.

step50 is the best policy this run produced. The update-60 row is included as a warning rather than a download: ten further updates took the policy below the model it started from. The cause is visible in the episode statistics - by update 60 the agent terminated voluntarily in 98 % of episodes after 18 tool calls, against 51 at update 50, so truncation fell to 2 % because episodes stopped early, not because the policy had become efficient. Training reward over those updates fell from about 0.55 to 0.35 and the in-training held-out evaluation from 46.9 % to 33.6 %, consistent with the benchmark.

An earlier negative-warmup run (nwf0917a, 40 updates) scored 47.80 % pass@1 at 98,304 context / 100 turns, the lowest of the six policies in that sweep; it is a different run and is published separately.

Conversion details

Exported with slime's tools/convert_torch_dist_to_hf.py (--vocab-size 248320 -a), pointing at each iter_* directory directly, so the marker file is never consulted and a base-weight fallback cannot occur. Every export was checked before upload: the full 738-tensor key set of the base model, matching shapes, and at least one sampled weight matrix differing from the base model.

• The vision tower is byte-identical to Qwen/Qwen3.5-4B; training updated the language model only.
• 48 tensors are stored in bfloat16 where the base model uses float32: linear_attn.A_log and linear_attn.norm.weight on each of the 24 layers. Training held every parameter in bfloat16, and that is the precision the inference server served during training and evaluation.
• At a learning rate of 1e-6 a single update moves a large matrix by roughly one bfloat16 ulp on 16–39 % of its elements, so even five updates apart these checkpoints look nearly identical tensor-by-tensor. They are distinct policies, not duplicates.