Llama-3-8B-D-STEER-OpenHermes-Reverse-DPO-Adapter

D-STEER · Llama-3-8B OpenHermes — Reverse-DPO adapter

⚠️ Safety notice — read before use

• This is an experimental reverse-preference adapter.
• It was intentionally trained to move behaviour away from harmlessness, by running preference optimization with the chosen and rejected responses swapped.
• It may increase harmful enablement relative to the unsteered base checkpoint.
• It must not be deployed as a general assistant or as a safety model, in any product, service, or user-facing system.
• It is released for controlled AI-safety research only.

D-STEER is a research programme on activation steering for safety behaviour in language models: whether a single direction in activation space, estimated from the difference between a preference-tuned checkpoint and its instruction-tuned starting point, can install or remove harmlessness behaviour at inference time.

Why this adapter exists

D-STEER asks whether removing safety behaviour with an activation-steering direction resembles removing it by training. This adapter is the training-based comparison: the same preference data, with the preference labels reversed.

It is used as a directional comparator — a reference point for the sign and rough scale of a de-alignment effect. It is not offered as proof that reverse training and activation steering are equivalent mechanisms, and no equivalence is claimed. Measurements in the D-STEER analyses treat this comparison as secondary and observational: the adapter differs from a steered model in training as well as in mechanism, so a difference in magnitude is not attributable to steering alone.

Model details

• Adapter type: LoRA (PEFT), r=16, alpha=32, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Base checkpoint: samarthraina/Llama-3-8B-D-STEER-OpenHermes-IT
• Base foundation model: meta-llama/Meta-Llama-3-8B
• Preference data: Anthropic HH-RLHF, with chosen and rejected responses swapped
• Training: 1 epoch, 2510 optimizer steps, learning rate 5e-7, effective batch size 16, DPO beta 0.1, bfloat16 compute
• Stored precision: LoRA tensors in float32 (448 tensors)

Full hyperparameters are in training_args.json; the optimizer trace is in trainer_state.json.

Requires transformers >= 5.0. The tokenizer in this repository is stored in the current tokenizers-backend format. Loading it with transformers 4.x raises Tokenizer class TokenizersBackend does not exist.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

ADAPTER = "samarthraina/Llama-3-8B-D-STEER-OpenHermes-Reverse-DPO-Adapter"
BASE = "samarthraina/Llama-3-8B-D-STEER-OpenHermes-IT"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

# optional: fold the adapter into the base weights
# model = model.merge_and_unload()

The base checkpoint and this adapter share the same tokenizer (128,257 tokens, identical tokenizer.json), so no embedding resize is needed here.

Intended use

Controlled AI-safety research only: studying de-alignment, comparing training-based and inference-time interventions, and red-teaming evaluation pipelines in a contained setting.

Out-of-scope use

Deployment as an assistant; any user-facing system; any safety-critical application; generating harmful content for distribution; any use inconsistent with the Meta Llama 3 Acceptable Use Policy.

Limitations and safety

• Behaviour is deliberately shifted away from harmlessness. Expect increased compliance with harmful requests.
• The effect is a shift in tendency, not a guarantee in either direction; outputs are not reliably harmful, safe, or accurate.
• Evaluated only in English, on research prompt sets, as a research comparator.
• Handle generations as untrusted research data.

Provenance and verification

License and attribution

Built with Meta Llama 3.

This model is a derivative of meta-llama/Meta-Llama-3-8B (revision 8cde5ca8380496c9a6cc7ef3a8b46a0372a1d920) and is distributed under the Meta Llama 3 Community License. Use of this model is subject to that agreement and to Meta's Acceptable Use Policy. By using these weights you agree to both.

• Meta Llama 3 Community License: https://llama.meta.com/llama3/license/
• Meta Llama 3 Acceptable Use Policy: https://llama.meta.com/llama3/use-policy/
• Base foundation model: meta-llama/Meta-Llama-3-8B

This repository does not contain a foundation model trained from scratch. It is a research checkpoint derived from Meta Llama 3 8B.