Downloads · 30 days
0
GPT2-Small-SAEs
jbloom/GPT2-Small-SAEs
GPT2-Small-SAEs is a machine learning model from jbloom. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
We trained 12 Sparse Autoencoders on the Residual Stream of GPT2-small. Each of these contains ~ 25k features as we used an expansion factor of 32 and the residual stream dimension of GPT2 has 768 dimensions. We train…
Access
Public
Updated Mar 9, 2024
Repo size
2 GB
Likes
17
Public
Repo makeup
Click a slice to open those files.
.pt2 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Jan 31, 2024
- Updated
- Mar 9, 2024
- SHA
- 3a66e186
Datasets
Model card
From the Hugging Face model README
We trained 12 Sparse Autoencoders on the Residual Stream of GPT2-small. Each of these contains ~ 25k features as we used an expansion factor of 32 and the residual stream dimension of GPT2 has 768 dimensions. We trained with an L1 coefficient of 8e-5 and learning rate of 4e-4 for 300 Million tokens, storing a buffer of ~500k tokens from OpenWebText which is refilled and shuffled whenever 50% of the tokens are used. To avoid dead neurons, we use ghost gradients. Our encoder/decoder weights are untied but we do use a tied decoder bias initialized at the geometric median per Bricken et al.
- License
- mit
- Languages
- en
- Created
- Jan 31, 2024
- Updated
- Mar 9, 2024