Downloads · 30 days
0
sae-llama-3-8b-32x-v2
EleutherAI/sae-llama-3-8b-32x-v2
sae-llama-3-8b-32x-v2 is a machine learning model from EleutherAI. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
This is a set of sparse autoencoders (SAEs) trained on the residual stream of Llama 3 8B using the 10B sample of the RedPajama v2 corpus, which comes out to roughly 8.5B tokens using the Llama 3 tokenizer. The SAEs ar…
Access
Public
Updated Jul 16, 2024
Repo size
73 GB
Likes
17
Public
Repo makeup
Click a slice to open those files.
.safetensors73 GB · 100%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Jul 12, 2024
- Updated
- Jul 16, 2024
- SHA
- d31d3992
Model card
From the Hugging Face model README
This is a set of sparse autoencoders (SAEs) trained on the residual stream of Llama 3 8B using the 10B sample of the RedPajama v2 corpus, which comes out to roughly 8.5B tokens using the Llama 3 tokenizer. The SAEs are organized by layer, and can be loaded using the EleutherAI sae library.
The layers.24 SAE in this repo has finished training on all 8.5B tokens of the RedPajama V2 sample. With the sae library installed, you can access it like this:
from sae import Sae
sae = Sae.load_from_hub("EleutherAI/sae-llama-3-8b-32x-v2", hookpoint="layers.24")
The rest of the SAEs are early checkpoints of an ongoing training run which can be tracked here. They will be updated as the training run progresses. The last upload was at 7,000 steps.
- Library
- transformers
- License
- mit
- Languages
- en
- Created
- Jul 12, 2024
- Updated
- Jul 16, 2024