Downloads · 30 days
0
sae-encoders
nileshsarkar-ai/sae-encoders
sae-encoders is a machine learning model from nileshsarkar-ai. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
Access
Public
Updated May 16, 2026
Repo size
29 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json172 MB · 84%
At a glance
- Library
- pytorch
- License
- apache-2.0
- Access
- Public
- Created
- May 16, 2026
- Updated
- May 16, 2026
- SHA
- 5bfa84aa
Base models
Model card
From the Hugging Face model README
SAE Encoders — codebase + experiment artifacts
Codebase, results, figures, and paper artifacts for the COLM 2026 work on Sparse Autoencoder scaling laws (Erdős AI Labs).
Headline figures
 |  |
| Toy-model loss-floor curves (exp1, main paper result) | Per-layer SAE metric heatmap (run_trial_2) |
The model weights themselves live in four sister Hugging Face repos (this repo holds the code + small artifacts; weights total ~195 GB so they're split out):
| Hugging Face repo | Contents | Distinguishing setting |
|---|---|---|
🔗 pythia-410m-saes-x32-l1-adaptive | 144 SAE checkpoints (24 layers × 6 snapshots) | L1 = 5e-4 adaptive, target L0 ≈ 150 |
🔗 pythia-410m-saes-x32-l1-3e-4-fixed | 144 SAE checkpoints | L1 = 3e-4 fixed |
🔗 pythia-410m-saes-x32-l1-8e-5-fixed | 144 SAE checkpoints | L1 = 8e-5 fixed |
🔗 erdos-forge-7b-qlora | Qwen2.5-Coder-7B QLoRA adapters (FORGE) | unrelated CAD-reasoning project |
Everything in this repo is also mirrored on GitHub at github.com/nileshsarkar-ai/Erdos-AI-Labs.
Repository structure
| Path | What it is |
|---|---|
run_trial_2/ | Trial 2 — first full Pythia-410M SAE training run with adaptive L1 (see run_trial_2/README.md) |
run_trial_3/ | Trial 3 — Trial-2 successor with dead-feature resampling, L1 warmup, ghost-grad fixes (see run_trial_3/README.md) |
run_exp_2_t1/ | Exp 2, run 1 — fixed L1 = 3e-4 sweep point (see run_exp_2_t1/README.md) |
run_exp_2_t2/ | Exp 2, run 2 — fixed L1 = 8e-5 sweep point + heatmap visualization suite (see run_exp_2_t2/README.md) |
exp1/ | Toy-model validation — KD minimum-width theorem (see exp1/README.md) |
exp1_results_COLM/ | Paper-ready figures + tables from exp1 (10 figures, 300 dpi PNG+PDF, JSON tables) |
exp1_results_1k/ | Earlier 1k-config sweep of exp1 |
older/ | Archived earlier scripts and SAE runs (Pythia-110M / single-layer experiments) |
ERDOS/ | FORGE project — constraint-graph CAD-reasoning system (separate from SAE work) |
colm-docs/ | Paper PDFs (deep-dive, experiment plan, full paper draft) |
Quick start
- Pull the checkpoints for one of the runs:
hf download nileshsarkar-ai/pythia-410m-saes-x32-l1-adaptive \ --local-dir ./run_trial_2/checkpoints - Re-run the measurement phase without retraining:
cd run_trial_2 python run_trial_2.py --phase measure - Regenerate figures:
python visualize_heatmap.py
Hardware used
NVIDIA A100 80 GB PCIe · Intel Xeon 42-core Cascade Lake · 62 GB RAM Each full training run (300 M tokens, 24 layers in parallel) took ~24 h.
Contact
For questions: Nilesh Sarkar — [email protected]
Citation
If you build on this work, please cite the COLM 2026 paper (see colm-docs/).
- Library
- pytorch
- License
- apache-2.0
- Created
- May 16, 2026
- Updated
- May 16, 2026