Downloads · 30 days
0
GPT2-Small-OAI-v5-128k-resid-post-SAEs
jbloom/GPT2-Small-OAI-v5-128k-resid-post-SAEs
GPT2-Small-OAI-v5-128k-resid-post-SAEs is a machine learning model from jbloom. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
OpenAI's GPT2-Small SAEs reformatted for easy loading from SAE Lens.
Access
Public
Updated Jul 1, 2024
Repo size
9.7 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors9.7 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Jun 30, 2024
- Updated
- Jul 1, 2024
- SHA
- a88f6e1e
Model card
From the Hugging Face model README
OpenAI's GPT2-Small SAEs reformatted for easy loading from SAE Lens.
Links
import torch
from transformer_lens import HookedTransformer
from sae_lens import SAE, ActivationsStore
torch.set_grad_enabled(False)
model = HookedTransformer.from_pretrained("gpt2-small")
sae, cfg, sparsity = SAE.from_pretrained(
"gpt2-small-resid-post-v5-128k", # to see the list of available releases, go to: https://github.com/jbloomAus/SAELens/blob/main/sae_lens/pretrained_saes.yaml
"blocks.11.hook_resid_post" # change this to another specific SAE ID in the release if desired.
)
# For loading activations or tokens from the training dataset.
activation_store = ActivationsStore.from_sae(
model=model,
sae=sae,
streaming=True,
# fairly conservative parameters here so can use same for larger
# models without running out of memory.
store_batch_size_prompts=8,
train_batch_size_tokens=4096,
n_batches_in_buffer=4,
device=device,
)
- License
- mit
- Created
- Jun 30, 2024
- Updated
- Jul 1, 2024