Downloads · 30 days
9
19% of all-time downloads
siddharthmb/2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B
2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B is a machine learning model from siddharthmb. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
A LoRA DPO adapter for Qwen/Qwen3-8B trained to prefer rejecting a negotiation package that pays the acting seat less than its own walk-away threshold, and accepting one that pays above it.
Downloads · 30 days
9
19% of all-time downloads
All-time downloads
47
Public
Repo size
22.7 GB
Likes
0
Public
Click a slice to open those files.
.safetensors19.2 GB · 82%
From the Hugging Face model README
A LoRA DPO adapter for Qwen/Qwen3-8B trained to prefer rejecting a negotiation package that pays the acting seat less than its own walk-away threshold, and accepting one that pays above it.
It works as a preference and fails as an intervention. It reaches held-out DPO reward accuracy 0.78–0.85, and it lowers the target behaviour in fresh rollouts — but a label-shuffled control adapter trained identically lowers it just as much, so none of the behavioural gain is attributable to the polarity signal. Read this card before using the adapter for anything: the honest summary is that it is a negative result, published because the control is what makes it one.
Companion: 2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B (the control arm — same steps, LR, seed, LoRA config and data volume, polarity randomized). All evaluation data, per-pair scores, pair sets and transcripts: 2026.RA.SelfHarm-Polarity-Arm.
75 preference pairs extracted from Qwen3-8B's own rollouts of a five-party scorable negotiation. At a real decision point where the model signed a package worth less than walking away, the pair is accept P against reject P — same offer, same serializer, no prose on either side, so the only thing separating chosen from rejected is the sign of (package value − own threshold). Sound accepts (packages comfortably above threshold) are included with the polarity running the other way, so an "always reject" policy scores 0.5 by construction rather than being caught afterwards.
| endpoint | untrained Qwen3-8B | this adapter | shuffled-label control | Δ this − untrained (paired) |
|---|---|---|---|---|
| self-harm rate | 0.047 [0.033, 0.062] | 0.032 [0.020, 0.046] | 0.031 [0.021, 0.042] | −0.0151 [−0.0285, −0.0026] |
| deal rate | 0.481 [0.374, 0.592] | 0.357 [0.256, 0.464] | 0.423 [0.317, 0.536] | −0.1243 [−0.2345, −0.0147] |
| non-action rate | 0.020 [0.015, 0.027] | 0.051 [0.042, 0.062] | 0.027 [0.020, 0.034] | +0.0311 [0.0186, 0.0443] |
| malformed emissions | 0.015 [0.010, 0.021] | 0.041 [0.034, 0.049] | 0.021 [0.015, 0.027] | +0.0260 [0.0172, 0.0354] |
| mean normalized surplus | 0.166 [0.115, 0.223] | 0.130 [0.085, 0.179] | 0.161 [0.111, 0.216] | −0.0359 [−0.0823, 0.0085] |
Against its control, paired on the same instances: self-harm +0.0013 [−0.0099, +0.0141] (nothing), non-action +0.0246 [0.0122, 0.0378] and malformed +0.0198 [0.0113, 0.0286] (worse). The self-harm reduction is not attributable to the polarity label; the closure and format damage is.
Why the numbers mean what they say. Intervals are percentile bootstraps over game instances as clusters (24 of them), never over rollout seeds, which share a score-sheet draw. Differences are paired: each resample draws instances jointly across arms, which is valid because all three arms played the same bank. A deal-rate guard of ±0.10 was registered in writing before the run, because the cheap way to stop signing bad packages is to stop signing anything — this adapter fails it at −0.124, and its rise in non-action says the same thing mechanically: a seat that emits no parseable action cannot commit this error.
| metric (held-out pairs) | this adapter | control |
|---|---|---|
| DPO reward accuracy | 0.78 – 0.85 | 0.49 – 0.63 |
| implicit-reward margin | +0.42 | ~0.00 |
| final train loss | 0.556 | 0.653 |
Balanced accuracy on the 41 held-out pairs is 0.500 with prefers_reject_rate 0.000 — identical to the untrained model, which never prefers reject either. The underlying quantity did move: training shifted the log-probability difference toward the correct action by +2.50 nats on average (control: +0.25), in the right direction on 29 of 41 pairs, but flipped none, because the median pair sits 14.4 nats on the wrong side of the boundary. A large, generalizing reward improvement that changes no argmax decision is the substance of this result.
(Only balanced accuracy is quotable on that split: it is 26 self-harm against 15 sound accepts, so raw accuracy hands 0.634 to an always-reject policy.)
python p4_train_dpo.py \
--run-name selfharm_polarity_trained_v1 \
--dataset-dir /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/datasets/selfharm_trained \
--pair-models Qwen3-8B \
--include-final-votes --include-private \
--regret-weighting none \
--pair-contrast as_built \
--learning-rate 5e-6 --epochs 4 --grad-accum 4 --per-device-batch-size 1 \
--min-eval-pairs 0 --eval-steps 12 --save-steps 12 --save-total-limit 6 --seed 0 \
--merged-name Qwen3-8B-selfharm-trained \
--wandb-project rational_agents_selfharm --wandb-mode offline
75 pairs × 4 epochs = 72 optimizer steps at effective batch 4. LoRA r=32, α=64, dropout 0.05, on attention and MLP projections (the behaviour is a value computation over a score sheet, and that arithmetic lives in the MLPs). bf16, gradient checkpointing, max length 4096 (no pair was dropped: p50 2,505 tokens, p99 3,772). The reference policy is the adapter-disabled base model, so no second copy of the weights is held.
--regret-weighting none is mandatory for this dataset rather than a preference: the pairs have no severity notion — the preference is a sign, not a magnitude — so the pipeline stamps a uniform severity purely to satisfy the trainer's reporting, and any weighting scheme would silently turn that constant into a fake data-driven weighting.
Rebuild the dataset first with selfharm_slice.py --from-run <8B rollout cell> and selfharm_dpo_prepare.py; see the dataset card for the full pipeline.
Logged in offline mode — the compute nodes have no outbound egress — so there are no online run URLs. Syncable run directories sit beside the adapter on the cluster.
adapter_model.safetensors, adapter_config.json: the LoRA itself (87M trainable parameters). train_config.json and data_summary.json record the exact hyperparameters, the data mix and per-split token quantiles./nlp/scr/siddharth/ii_mats/rational_agents/p4_pilot_dpo/selfharm_polarity_trained_v1/, archived in full into this repository on 2026-10-10 (see "Full run archive" below); the local directory now holds only a pointer file./nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/three_arm_verdict.{json,md}/nlp/scr/siddharth/ii_mats/rational_agents/selfharm_eval_trained_v2/, /nlp/scr/siddharth/ii_mats/transcripts/rational_agents/selfharm_eval_v2bank/trained.jsonlexperiments/rational_agents/sbatch_selfharm/logs/ (training job 16601191; the preempted first launch 16600808; polarity readout 16601200; rollout post-processing 16601214; trained-arm rollouts 16601192 to 16601196)A LoRA adapter cannot be seated directly by the evaluation harness this was built for; merge it first (p4_train_dpo.py --merge-adapter, which merges onto the same base string the run trained from, so the wrong-base failure mode — a model that is neither base nor adapter, and that no downstream metric reveals — cannot occur). Scoring tools that read adapter_config.json load it directly with no merge.
The root-level files above are the adapter as first published on 2026-08-10. On 2026-10-10 the entire local run directory was archived into this repository at its original relative paths, so the checkpoints with optimizer state and the merged full-weight model the rollout fleet actually seated are here too. The local run name is selfharm_polarity_trained_v1; "self-harm" in this programme is the bargaining-game term defined above (a seat signing or tabling a package that pays it less than its own walk-away threshold), nothing else.
adapter/ the same final LoRA (step 76) as the root files: adapter_model.safetensors (349 MB), adapter_config.json, tokenizer files, training_args.bin, TRL stub README
checkpoints/checkpoint-{24,36,48,60,72,76}/ LoRA + optimizer.pt (699 MB) + scheduler.pt + rng_state.pth + trainer_state.json (full log history) + tokenizer (checkpoint-12 was rotated out by --save-total-limit 6)
checkpoints/README.md TRL-generated stub
Qwen3-8B-selfharm-trained/ the adapter merged into Qwen/Qwen3-8B: bf16 full weights, 4 safetensors shards (16.4 GB), tokenizer, config; this is what the 120-episode trained-arm rollouts (`selfharm_eval_trained_v2`) played
train_config.json, data_summary.json same content as the root copies (resolved config with code provenance: parent repo sha 3c72b1e, interlens 17f34b2 v0.1.60; per-split pair statistics)
merge_config.json the merge invocation's resolved config (`--merge-adapter --merged-name Qwen3-8B-selfharm-trained`)
train.log log of the last invocation written to the run dir (the merge)
Training-side numbers from checkpoints/checkpoint-76/trainer_state.json: 76 optimizer steps over 4 epochs (75 pairs at effective batch 4); train loss 0.696 at step 5 and 0.390 at step 75; held-out DPO reward accuracy 0.610 (step 12), 0.683 (24), 0.878 (36), 0.805 (48), 0.854 (60), 0.805 (72), implicit-reward margin rising to +0.42. The run is slurm job 16601191 (the clean restart; the preempted first launch 16600808 is archived in 2026.RA.P4-Pilot-DPO-Misc-Runs under selfharm_polarity_trained_v1_preempted_16600808/). The control arm's full run is archived the same way in 2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B.
Loading the merged model directly, or a checkpoint:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
REPO = "siddharthmb/2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B"
tok = AutoTokenizer.from_pretrained(REPO, subfolder="Qwen3-8B-selfharm-trained")
model = AutoModelForCausalLM.from_pretrained(REPO, subfolder="Qwen3-8B-selfharm-trained", torch_dtype="bfloat16")
# a LoRA checkpoint on the base:
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, REPO, subfolder="checkpoints/checkpoint-60")
Whole tree: hf download siddharthmb/2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B --local-dir selfharm_polarity_trained_v1.
Local provenance: Archived from /nlp/scr/siddharth/ii_mats/rational_agents/p4_pilot_dpo/selfharm_polarity_trained_v1 on 2026-10-11 by the MATS archive job 17819017 of the transcoder-adapters cleanup (report claude_updates/memory/reports/mats_archive.md in that repository). The run directory was uploaded unchanged (upload_large_folder), so every checkpoint, config and log keeps its original relative path. Every file name and every text file was scanned for credentials before upload (Hub, Anthropic, OpenAI, AWS and W&B key shapes, private-key headers, credential URLs, paths of the secrets store); the report lists what was excluded. The local copy was deleted after every uploaded file was verified on the Hub by name, size and hash, and a <name>.txt breadcrumb naming this repository was left beside it. Collection: https://huggingface.co/collections/siddharthmb/2026-mats-rational-agents-6acc1dd5623e8e00176b80a8; workspace index: https://huggingface.co/datasets/siddharthmb/2026.RA.Workspace-Index.