Downloads · 30 days
15
2% of all-time downloads
zhengxuanzenwu/intervenable_honest_llama2_chat_7B
intervenable_honest_llama2_chat_7B is a machine learning model from zhengxuanzenwu. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
This repo ~0.14MB contains everything you need to reproduce the inference-time intervention paper. The runtime is about 1.8x comparing to regular generate, but there is minimum memory cost.
Downloads · 30 days
15
2% of all-time downloads
All-time downloads
626
Public
Repo size
141 KB
Likes
3
Public
Click a slice to open those files.
.bin141 KB · 95%
From the Hugging Face model README
This repo ~0.14MB contains everything you need to reproduce the inference-time intervention paper. The runtime is about 1.8x comparing to regular generate, but there is minimum memory cost.
To use the repo, run install of pyvene from ToT:
pip install git+https://github.com/stanfordnlp/pyvene.git
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import pyvene as pv
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-chat-hf",
torch_dtype=torch.bfloat16,
).to("cuda")
pv_model = pv.IntervenableModel.load(
# the activation diff ~0.14MB
"zhengxuanzenwu/intervenable_honest_llama2_chat_7B",
model,
)
q = "What's a cure for insomnia that always works?"
prompt = tokenizer(q, return_tensors="pt").to("cuda")
_, iti_response_shared = pv_model.generate(
prompt, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(iti_response_shared[0], skip_special_tokens=True))