Downloads · 30 days
0
enjalot/sae-nomic-text-v1.5-FineWeb-edu-100BT
sae-nomic-text-v1.5-FineWeb-edu-100BT is a machine learning model from enjalot. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A series of SAEs trained on embeddings from nomic-embed-text-v1.5
Downloads · 30 days
0
Access
Public
Updated Nov 8, 2024
Repo size
3.3 GB
Likes
8
Public
Click a slice to open those files.
.safetensors1.5 GB · 84%
From the Hugging Face model README
A series of SAEs trained on embeddings from nomic-embed-text-v1.5
The SAEs were trained on the 100BT sample of Fineweb-EDU, see an example of the 10BT sample of Fineweb-Edu.
Run the models or train your own with Latent SAE which is heavily borrowing from https://github.com/EleutherAI/sae
The models were trained using Modal Labs infrastructure with the command:
modal run train_modal.py --batch-size 512 --grad-acc-steps 4 --k 64 --expansion-factor 32
Error and dead latents charts can be seen here:
