matryoshka-width-ladder-gemma2-2b

matryoshka-width-ladder-gemma2-2b

A width ladder of Matryoshka BatchTopK sparse autoencoders (SAEs) trained with SAELens 6.43.0 on gemma-2-2b residual-stream activations (blocks.12.hook_resid_post, d_in 2304): widths {16384, 32768, 65536, 131072} at k=100, 3 random seeds per width (12 runs). Each run trained on 300M tokens of pretokenized Pile-uncopyrighted (chanind/pile-uncopyrighted-gemma-1024-abbrv-2B, context 1024), with 11 training snapshots saved from 0 to 300M tokens.

Matryoshka training uses nested prefix dictionaries at 1/16 and 1/4 of the full width plus the full dictionary (exact levels in each run's runner_cfg.json under sae.matryoshka_widths). Runs are saved in SAELens JumpReLU inference format (calibrated threshold), the standard export for BatchTopK-family training.

Layout

panel/gemma-2-2b/mat-k100-w<width>-L12/seed-<n>/ with:

• cfg.json, sae_weights.safetensors, sparsity.safetensors — SAELens-loadable final SAE
• runner_cfg.json — full training configuration
• train_summary.json — per-snapshot dead-feature fraction, mean log sparsity, implied L0
• snapshots/step-<step>-tokens-<tokens>/ — 11 SAELens-loadable snapshots (0 to 300M tokens)

Loading

from huggingface_hub import snapshot_download
from sae_lens import SAE

run = "panel/gemma-2-2b/mat-k100-w65536-L12/seed-0"
local = snapshot_download("chanind-goodfire/matryoshka-width-ladder-gemma2-2b", allow_patterns=[f"{run}/*"])
sae = SAE.load_from_disk(f"{local}/{run}")

Snapshot dirs load the same way. upload_manifest.json lists every file with its size and sha256.