Downloads · 30 days
0
saes-qwen2.5-7b-instruct
andyrdt/saes-qwen2.5-7b-instruct
saes-qwen2.5-7b-instruct is a machine learning model from andyrdt. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Residual stream SAEs for Qwen2.5-7B-Instruct.
Access
Public
Updated May 26, 2025
Repo size
105 GB
Likes
3
Public
Repo makeup
Click a slice to open those files.
.pt105 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- May 26, 2025
- Updated
- May 26, 2025
- SHA
- c37e53c4
Model card
From the Hugging Face model README
Residual stream SAEs for Qwen2.5-7B-Instruct.
These SAEs were trained using a blend of chat (lmsys/lmsys-chat-1m) and pretraining data (monology/pile-uncopyrighted), and also a small amount of emergent misalignment data.
Each SAE is trained using BatchTopK. For each layer, we train 4 SAEs, with k=32,64,128,256.
For more training details, see https://github.com/andyrdt/dictionary_learning/tree/andyrdt/qwen.
Note: the first 8 tokens of each sample are excluded from training, and additionally activations with large outlier norms (>10x median norm of a batch) are filtered out.
- License
- apache-2.0
- Created
- May 26, 2025
- Updated
- May 26, 2025