2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B

2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B

The control arm, published because it is the finding. This LoRA adapter was trained on the same 75 negotiation decision points as 2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B, with the preference label randomized — identical steps, learning rate, seed, LoRA configuration and data volume; only the label carries (or fails to carry) information. Its chosen side is correct on 39 of 75 pairs (52%, chance).

It exists to answer one question: when the real arm lowers the target behaviour, is that the polarity signal or just DPO on these prompts? The answer is the second. This control reduces the target behaviour as much as the real arm does, which is what converts a publishable-looking 1.5-point improvement into a negative result.

All evaluation data, per-pair scores, pair sets and transcripts: 2026.RA.SelfHarm-Polarity-Arm.

The target behaviour

Self-harm in a five-party scorable negotiation: a seat accepting or proposing a package that pays it strictly less than its own private walk-away threshold. Checkable from the seat's own prompt; rejecting such a package is correct play and never counts.

Performance: 120 fresh episodes on 24 game instances neither arm saw

This label-free arm moves the primary endpoint slightly more than the labelled one (−0.0163 against −0.0151; the direct paired contrast is +0.0013 [−0.0099, +0.0141], i.e. nothing), and it does so without the labelled arm's costs: it passes the pre-registered deal-rate guard that the labelled arm fails, and its rises in non-action and malformed emissions are not distinguishable from zero.

Intervals are percentile bootstraps over game instances as clusters; differences are paired, resampling instances jointly across arms, which is valid because all three arms played the same bank.

Confirmation that it learned nothing about polarity

So the two arms are cleanly separated at the preference level and indistinguishable at the behavioural one. That gap is the result.

Training command

Identical to the real arm except for the dataset directory:

python p4_train_dpo.py \
  --run-name selfharm_polarity_control_v1 \
  --dataset-dir /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/datasets/selfharm_control \
  --pair-models Qwen3-8B \
  --include-final-votes --include-private \
  --regret-weighting none \
  --pair-contrast as_built \
  --learning-rate 5e-6 --epochs 4 --grad-accum 4 --per-device-batch-size 1 \
  --min-eval-pairs 0 --eval-steps 12 --save-steps 12 --save-total-limit 6 --seed 0 \
  --merged-name Qwen3-8B-selfharm-control \
  --wandb-project rational_agents_selfharm --wandb-mode offline

The control pair file is produced by the same extractor as the training set (selfharm_slice.py, --shuffle-polarity path), so the two arms cannot drift apart in anything but the label. LoRA r=32, α=64, dropout 0.05, attention + MLP projections, bf16, 72 optimizer steps.

Limitation worth stating

This is a near-null arm, not a null arm: randomizing polarity on half the pairs leaves 52% of chosen sides correct, so a residual few points of signal cannot be formally excluded. Empirically its held-out reward accuracy sits at chance, which is the strongest available evidence that it carries none.

Weights & Biases

Offline mode (compute nodes have no egress); no online run URLs. Syncable run directories sit beside the adapter on the cluster.

Cluster paths

• Adapter and merged model: formerly /nlp/scr/siddharth/ii_mats/rational_agents/p4_pilot_dpo/selfharm_polarity_control_v1/, archived in full into this repository on 2026-10-10 (see "Full run archive" below); the local directory now holds only a pointer file.
• Evaluation results: /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/three_arm_verdict.{json,md}
• Rollout cell and transcripts: /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_eval_control_v2/, /nlp/scr/siddharth/ii_mats/transcripts/rational_agents/selfharm_eval_v2bank/control.jsonl
• Slurm logs: experiments/rational_agents/sbatch_selfharm/logs/ (training job 16600809; polarity readout 16601200; rollout post-processing 16601214)

Full run archive (added 2026-10-10)

The root-level files above are the adapter as first published on 2026-08-10. On 2026-10-10 the entire local run directory was archived into this repository at its original relative paths, so the checkpoints with optimizer state and the merged full-weight model the rollout fleet actually seated are here too. The local run name is selfharm_polarity_control_v1; "self-harm" in this programme is the bargaining-game term defined above (a seat signing or tabling a package that pays it less than its own walk-away threshold), nothing else.

adapter/                                   the same final LoRA (step 76) as the root files: adapter_model.safetensors (349 MB), adapter_config.json, tokenizer files, training_args.bin, TRL stub README
checkpoints/checkpoint-{24,36,48,60,72,76}/ LoRA + optimizer.pt (699 MB) + scheduler.pt + rng_state.pth + trainer_state.json (full log history) + tokenizer  (checkpoint-12 was rotated out by --save-total-limit 6)
checkpoints/README.md                      TRL-generated stub
Qwen3-8B-selfharm-control/                 the adapter merged into Qwen/Qwen3-8B: bf16 full weights, 4 safetensors shards (16.4 GB), tokenizer, config; this is what the 120-episode control-arm rollouts (`selfharm_eval_control_v2`) played
train_config.json, data_summary.json       same content as the root copies (resolved config with code provenance: parent repo sha 3c72b1e, interlens 17f34b2 v0.1.60; per-split pair statistics)
merge_config.json                          the merge invocation's resolved config (`--merge-adapter --merged-name Qwen3-8B-selfharm-control`)
train.log                                  log of the last invocation written to the run dir (the merge)

Training-side numbers from checkpoints/checkpoint-76/trainer_state.json: 76 optimizer steps over 4 epochs (75 shuffled-label pairs at effective batch 4); train loss 0.685 at step 5 and 0.632 at step 75; held-out DPO reward accuracy 0.390 (step 12), 0.317 (24), 0.390 (36), 0.488 (48), 0.488 (60), 0.634 (72), implicit-reward margin within +/-0.03 throughout, which is the "learned nothing about polarity" confirmation in the table above. The run is slurm job 16600809 (ran uninterrupted). The trained arm's full run is archived the same way in 2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B; its preempted first launch is in 2026.RA.P4-Pilot-DPO-Misc-Runs.

Loading the merged model directly, or a checkpoint:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
REPO = "siddharthmb/2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B"
tok = AutoTokenizer.from_pretrained(REPO, subfolder="Qwen3-8B-selfharm-control")
model = AutoModelForCausalLM.from_pretrained(REPO, subfolder="Qwen3-8B-selfharm-control", torch_dtype="bfloat16")
# a LoRA checkpoint on the base:
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, REPO, subfolder="checkpoints/checkpoint-60")

Whole tree: hf download siddharthmb/2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B --local-dir selfharm_polarity_control_v1.

Local provenance: Archived from /nlp/scr/siddharth/ii_mats/rational_agents/p4_pilot_dpo/selfharm_polarity_control_v1 on 2026-10-11 by the MATS archive job 17819017 of the transcoder-adapters cleanup (report claude_updates/memory/reports/mats_archive.md in that repository). The run directory was uploaded unchanged (upload_large_folder), so every checkpoint, config and log keeps its original relative path. Every file name and every text file was scanned for credentials before upload (Hub, Anthropic, OpenAI, AWS and W&B key shapes, private-key headers, credential URLs, paths of the secrets store); the report lists what was excluded. The local copy was deleted after every uploaded file was verified on the Hub by name, size and hash, and a <name>.txt breadcrumb naming this repository was left beside it. Collection: https://huggingface.co/collections/siddharthmb/2026-mats-rational-agents-6acc1dd5623e8e00176b80a8; workspace index: https://huggingface.co/datasets/siddharthmb/2026.RA.Workspace-Index.