openwebrl-ws-credit-beta0.4

OpenWebRL — within-state Scalar-RM credit assignment (ws-credit, β = 0.4)

Training checkpoints, optimizer state and per-iteration logs for an RL run that adds a within-state process reward to turn-level GRPO on the OpenWebRL web-agent task.

The point of the repository is reproducibility of a specific negative-or-positive result, and insurance: the run lives on node-local disk inside a Slurm allocation that has already been requeued once, taking the local disk with it.

The experiment in one block

At a labeled turn we draw K = 5 responses from the policy with the same prompt and sampling parameters, score all five with Piotr Teterwak's Scalar Action Reward Model, and execute candidate 0 — never the argmax, so the scores cannot influence behaviour and the trajectory stays on-policy:

d_k = 0                     if candidate k canonicalizes to the executed action
    = r_taken − r_k         otherwise            (k = the K−1 other candidates)
c_t = clip( mean_k d_k / τ , −1, +1 )
A_t = A + β · c_t

Everything else is the OpenWebRL reproduction unchanged: 48 groups × 5 rollouts, lr 1e-6, global batch 256, 2 PPO epochs, 15-step horizon, rollout temperature 0.8, dynamic filter, GPT-4.1 judge, starting from the OpenWebRL/OpenWebRL-4B-SFT checkpoint.

Layout

resume/iter_XXXXXXX/     Megatron distcp: model + optimizer + scheduler.
                         58 GB. This is what makes an exact resume possible.
resume/latest_checkpointed_iteration.txt
hf/iter_XXXXXXX/         converted HuggingFace weights, 9.1 GB. Policy only —
                         enough to evaluate or start a fresh run, not to resume one.
logs/ws_metrics.jsonl    one row per iteration, ~45 `ws/` metrics
logs/ws_credit_log.jsonl one row per trajectory: reward, A, per-turn action, the five
                         scores, same-action flags, c and A_t

Resuming training exactly

REPO=XunmeiLiu/openwebrl-ws-credit-beta0.4
N=$(huggingface-cli download $REPO resume/latest_checkpointed_iteration.txt --repo-type model \
      --quiet | xargs cat)
DST=/mnt/localssd/$USER/seleval/outputs/owrl_ws_beta0.4/ckpt
mkdir -p "$DST"
huggingface-cli download $REPO --repo-type model \
  --include "resume/iter_$(printf '%07d' $N)/*" --local-dir /tmp/wsdl
mv "/tmp/wsdl/resume/iter_$(printf '%07d' $N)" "$DST/"
echo "$N" > "$DST/latest_checkpointed_iteration.txt"

# the launcher picks the checkpoint up by itself
bash ~/seleval/start_ws_phase2.sh

run_train_8xh100.sh reads latest_checkpointed_iteration.txt, sets SLIME_LOAD_CHECKPOINT and SLIME_CKPT_STEP, and continues. Nothing else is needed.

Two environment gotchas that will otherwise cost hours on a fresh node:

• Megatron asserts numpy 1.x. Known-good pair: numpy 1.26.4 + scipy 1.16.3.
• These nodes ship /usr/local/cuda = 13.0 while the env is a cu12 wheel stack. slime builds the model through megatron.bridge from the HF config, which does not carry --attention-backend, so it arrives as auto, transformer_engine picks its fused kernel, and the fused loader dies with "Multiple libcudart libraries found: libcudart.so.12 and libcudart.so.13". Patch megatron/core/models/common/language_module/language_module.py::_set_attention_backend to treat auto as flash. Flash and fused are different kernels for identical mathematics. Setting NVTE_FUSED_ATTN=0 yourself does not work — Megatron sets those three flags from the config and asserts nobody else has.

What the run shows so far (iteration 44 of 60)

Mechanism, over all 45 iterations:

The decisive training-side gate — inside a mixed GRPO group, do the labeled rollouts that succeeded carry higher mean c than their failed siblings on the same task?

+0.0353 over 575 mixed groups, t = 2.75 (p = 0.0059), sign test 32/45 iterations positive (p = 0.0046), and it strengthens: +0.0214 over iterations 0–21 against +0.0483 over 22–44.

Training curve, first 10 iterations against the last 10: reward 0.430 → 0.504, all-failure groups 42.4 → 30.3, response length 325 → 368, truncation 1.0% → 1.4%.

A caveat worth stating: an offline, unpaired measurement over 274 evaluation tasks put the AUROC of trajectory-mean c for task success at 0.530. The paired within-group test above disagrees because the unpaired one is dominated by task difficulty and episode length; the paired comparison removes both by construction. Checkpoint evaluations against the plain-RL reproduction at the same iterations are the arbiter and are not in this README yet.

Provenance

Produced on an 8×H100 node (interna3m) by a CloudCode agent session, ses_e5fc8854969ffe3xWuV7yJqhv3. The Scalar RM is PTeterwak/OpenWebRL-4B-ScalarRM-LoRA; the policy starts from OpenWebRL/OpenWebRL-4B-SFT.