recipe-delta-sae-panel-66runs

recipe-delta-sae-panel-66runs

A panel of 66 sparse autoencoders (SAEs) trained with SAELens 6.43.0 on residual-stream activations of three base models: 22 training recipes x 3 random seeds, each run trained on 300M tokens of pretokenized Pile-uncopyrighted (chanind/pile-uncopyrighted-*, context 1024), with 11 training snapshots saved from 0 to 300M tokens.

Recipe grid

• gemma-2-2b (d_in 2304), 16 recipes:
  • BatchTopK sparsity sweep: k in {25, 50, 100, 120, 200, 400}, width 32768, layer 12
  • BatchTopK width sweep: width in {16384, 32768, 49152, 65536, 98304, 131072}, k=100, layer 12
  • BatchTopK layer sweep: blocks.{4,12,20}.hook_resid_post, k=100, width 32768
  • JumpReLU: target L0 in {50, 100, 200}, width 32768, layer 12
• llama-3.1-8b (meta-llama/Llama-3.1-8B, d_in 4096): BatchTopK k in {50, 100, 200}, width 32768, blocks.16.hook_resid_post
• pythia-160m (d_in 768): BatchTopK k in {25, 100, 400}, width 6144, blocks.6.hook_resid_post

BatchTopK runs are saved in SAELens JumpReLU inference format (calibrated threshold), the standard export for BatchTopK training.

Layout

panel/<model>/<recipe>/seed-<n>/ with:

• cfg.json, sae_weights.safetensors, sparsity.safetensors — SAELens-loadable final SAE
• runner_cfg.json — full training configuration
• train_summary.json — per-snapshot dead-feature fraction, mean log sparsity, implied L0
• snapshots/step-<step>-tokens-<tokens>/ — 11 SAELens-loadable snapshots (0 to 300M tokens)

Loading

from huggingface_hub import snapshot_download
from sae_lens import SAE

run = "panel/gemma-2-2b/btk-k100-w32768-L12/seed-0"
local = snapshot_download("chanind-goodfire/recipe-delta-sae-panel-66runs", allow_patterns=[f"{run}/*"])
sae = SAE.load_from_disk(f"{local}/{run}")

Snapshot dirs load the same way. upload_manifest.json lists every file with its size and sha256.