Downloads · 30 days
0
nileshsarkar-ai/pythia-410m-saes-x32-l1-3e-4-fixed
pythia-410m-saes-x32-l1-3e-4-fixed is a machine learning model from nileshsarkar-ai. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
Downloads · 30 days
0
Access
Public
Updated May 16, 2026
Repo size
58 GB
Likes
0
Public
Click a slice to open those files.
.pt58 GB · 100%
From the Hugging Face model README

Sparse Autoencoder (SAE) checkpoints trained on every residual-stream layer of
EleutherAI/pythia-410m, for the COLM SAE scaling-law experiments
(source code on GitHub,
full codebase on HF).
![]() | ![]() |
| Training curves across all 24 layers | Predicted vs measured loss floor |
| Base model | EleutherAI/pythia-410m |
| Layers covered | 0–23 (all 24) |
| SAE expansion factor | 32 → F = 32,768 dictionary features per layer |
| Hidden dim being modeled | 1024 (Pythia-410M residual stream) |
| L1 coefficient | 3e-4 (fixed) |
| Tokens trained | 300 M (PILE) |
| Snapshots per layer | 6 — at 50 M, 100 M, 150 M, 200 M, 250 M tokens, plus final |
| Total files | 144 .pt checkpoints (24 layers × 6 snapshots) |
sae_layer{LL}_{SNAPSHOT}.pt
Where LL is the layer index (00–23) and SNAPSHOT is one of
50M, 100M, 150M, 200M, 250M, final.
Examples:
sae_layer00_50M.ptsae_layer12_final.ptsae_layer23_250M.ptimport torch
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
repo_id="nileshsarkar-ai/pythia-410m-saes-x32-l1-3e-4-fixed",
filename="sae_layer12_final.pt",
)
state = torch.load(ckpt_path, map_location="cpu", weights_only=True)
| run | L1 coefficient | target |
|---|---|---|
| pythia-410m-saes-x32-l1-adaptive | 5e-4 (adaptive) | target L0 ≈ 150 |
| pythia-410m-saes-x32-l1-3e-4-fixed | 3e-4 | fixed |
| pythia-410m-saes-x32-l1-8e-5-fixed | 8e-5 | fixed |
Training script at
run_exp_2_t1/run_exp.py
in the source repo. Hardware: NVIDIA A100 80 GB PCIe.
python run_exp.py --phase train --num_tokens 300_000_000 --expansion 32 --l1_coeff 3e-4
run_exp_2_t1/results/.COLM_BACKUP_RESTORE.md.