Llama-3-8B-D-STEER-OpenHermes-DPO

D-STEER · Llama-3-8B OpenHermes — DPO checkpoint

This repository is the canonical D-STEER release of the matched Llama-3-8B OpenHermes checkpoint after harmlessness-oriented Direct Preference Optimization.

D-STEER is a research programme on activation steering for safety behaviour in language models: whether a single direction in activation space, estimated from the difference between a preference-tuned checkpoint and its instruction-tuned starting point, can install or remove harmlessness behaviour at inference time.

Matched pair

This checkpoint is matched with the IT checkpoint samarthraina/Llama-3-8B-D-STEER-OpenHermes-IT: same base foundation model, same instruction-tuning data, same tokenizer. The only difference is the added preference stage, which is what makes the pair usable for estimating a harmlessness direction.

Model details

• Base foundation model: meta-llama/Meta-Llama-3-8B, revision 8cde5ca8380496c9a6cc7ef3a8b46a0372a1d920
• Instruction-tuning data: OpenHermes-2.5
• Preference stage: Direct Preference Optimization on Anthropic HH-RLHF harmlessness data
• Format: self-contained merged causal language model (no adapter required)
• Stored precision: float16 (all 291 tensors)
• Architecture: Llama, 32 layers, hidden size 4096, vocabulary 128,257

Requires transformers >= 5.0. The tokenizer in this repository is stored in the current tokenizers-backend format. Loading it with transformers 4.x raises Tokenizer class TokenizersBackend does not exist.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "samarthraina/Llama-3-8B-D-STEER-OpenHermes-DPO"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.float16)
model.eval()

messages = [{"role": "user", "content": "Explain activation steering in two sentences."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Intended use

Released for AI-safety research: as the preference-tuned endpoint of a matched pair for estimating and evaluating harmlessness directions in activation space.

Limitations and safety

• Harmlessness training reduces but does not eliminate harmful completions. This is a research checkpoint, not a safety-hardened product.
• The preference stage targets harmlessness specifically; it is not a general capability, factuality, or robustness improvement, and it can increase refusal on benign requests.
• Not evaluated for production deployment.
• Inherits the limitations, biases, and knowledge cutoff of Meta Llama 3 8B, the OpenHermes-2.5 instruction data, and the HH-RLHF preference data.

Provenance and verification

PROVENANCE.json in this repository records the SHA-256 of every published file.

Matched checkpoint

The instruction-tuned counterpart is samarthraina/Llama-3-8B-D-STEER-OpenHermes-IT.

License and attribution

Built with Meta Llama 3.

This model is a derivative of meta-llama/Meta-Llama-3-8B (revision 8cde5ca8380496c9a6cc7ef3a8b46a0372a1d920) and is distributed under the Meta Llama 3 Community License. Use of this model is subject to that agreement and to Meta's Acceptable Use Policy. By using these weights you agree to both.

• Meta Llama 3 Community License: https://llama.meta.com/llama3/license/
• Meta Llama 3 Acceptable Use Policy: https://llama.meta.com/llama3/use-policy/
• Base foundation model: meta-llama/Meta-Llama-3-8B

This repository does not contain a foundation model trained from scratch. It is a research checkpoint derived from Meta Llama 3 8B.