D-STEER-Mistral-7B-OpenHermes-DPO-Adapter

D-STEER · Mistral-7B OpenHermes — harmlessness DPO adapter

Canonical D-STEER release of the harmlessness-oriented DPO adapter for teknium/OpenHermes-2.5-Mistral-7B.

D-STEER is a research programme on activation steering for safety behaviour in language models: whether a single direction in activation space, estimated from the difference between a preference-tuned checkpoint and its instruction-tuned starting point, can install or remove harmlessness behaviour at inference time.

This adapter supplies the preference-tuned endpoint of the second D-STEER model pair, used to check that findings are not specific to a single architecture.

Model details

• Base model: teknium/OpenHermes-2.5-Mistral-7B, revision 24c0bea14d53e6f67f1fbe2eca5bfe7cae389b33
• Base foundation model: mistralai/Mistral-7B-v0.1
• Instruction-tuning data (inherited from the base): OpenHermes-2.5
• Preference data: Anthropic HH-RLHF harmless-base
• Objective: harmlessness-oriented Direct Preference Optimization
• Adapter type: LoRA (PEFT), r=16, alpha=32, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning rate: 5e-6 · Effective batch size: 16 · DPO beta: 0.1 · bfloat16 compute
• Stored precision: LoRA tensors in float32; the shipped embed_tokens / lm_head in float16. The base model itself is bfloat16, which is what the snippet below loads.

Full hyperparameters are in training_args.json.

Requires transformers >= 5.0. The tokenizer in this repository is stored in the current tokenizers-backend format. Loading it with transformers 4.x raises Tokenizer class TokenizersBackend does not exist.

Important: tokenizer size

The training tokenizer in this repository has one additional token relative to the base model — a dedicated <|pad|> padding token:

The base embedding table must be resized to match before the adapter is attached, or loading fails on a shape mismatch. The snippet below does this correctly.

This repository ships the trained embed_tokens and lm_head weights at the resized [32003, 4096] shape alongside the LoRA tensors, so once you resize, the added token's values come from the adapter rather than from random initialisation.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

ADAPTER = "samarthraina/D-STEER-Mistral-7B-OpenHermes-DPO-Adapter"
BASE = "teknium/OpenHermes-2.5-Mistral-7B"
BASE_REVISION = "24c0bea14d53e6f67f1fbe2eca5bfe7cae389b33"

# 1. load the adapter's tokenizer (it carries the extra token)
tokenizer = AutoTokenizer.from_pretrained(ADAPTER)

# 2. load the base model at the pinned revision
model = AutoModelForCausalLM.from_pretrained(BASE, revision=BASE_REVISION, dtype=torch.bfloat16)

# 3. check the embedding table against the tokenizer
n_embed = model.get_input_embeddings().weight.shape[0]
print(f"base embeddings: {n_embed} | tokenizer: {len(tokenizer)}")

# 4. resize only if they disagree
if n_embed != len(tokenizer):
    model.resize_token_embeddings(len(tokenizer))

# 5. attach the LoRA adapter
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

# 6. optional: fold the adapter into the base weights
# model = model.merge_and_unload()

Intended use

Released for AI-safety research: as the preference-tuned endpoint of a matched pair for estimating and evaluating harmlessness directions in activation space.

Limitations and safety

• Harmlessness training reduces but does not eliminate harmful completions. This is a research adapter, not a safety-hardened product.
• The preference stage targets harmlessness specifically; it can increase refusal on benign requests and is not a general capability or factuality improvement.
• Not evaluated for production deployment.
• Inherits the limitations and biases of the base model, the OpenHermes-2.5 instruction data, and the HH-RLHF preference data. English-centric.

Provenance and verification

License

Apache-2.0, inherited from the base model teknium/OpenHermes-2.5-Mistral-7B (Apache-2.0), which is itself derived from mistralai/Mistral-7B-v0.1 (Apache-2.0). This adapter contains no Llama-derived weights.