sae-SmolLM2-1.7B-layer17-32x

SAE for SmolLM2-1.7B, layer 17

A standard (TopK) sparse autoencoder trained with sparsify on the residual stream of HuggingFaceTB/SmolLM2-1.7B at layer 17 (0-indexed, out of 24 layers — roughly 3/4 of the way through the model).

Training details

• Dataset: EleutherAI/SmolLM2-135M-10B, one full epoch (19,044 steps)
• Expansion factor: 32x (65,536 latents, d_in=2048)
• k: 32
• Optimizer: Adam, lr=1e-4 (see train_config.json for the full training config)
• Final FVU: 0.0172, dead latents: 0%

See train_config.json for the complete sparsify.TrainConfig used.

Usage

from sparsify import Sae

sae = Sae.load_from_hub("EleutherAI/sae-SmolLM2-1.7B-layer17-32x")

Automated interpretability

Explanations were generated and scored with delphi automated interpretability over a growing sample of latents (currently the first 900 latent indices, of which 883 cleared the ≥200 activating-example threshold and were scored).

Protocol

• Activations cached over 200M tokens of the training distribution (EleutherAI/SmolLM2-135M-10B, BOS-filtered). Whole-SAE at this budget: 0 dead latents, and 98.4% of latents fire ≥200× (64,456 / 65,536).
• Explainer & scorer: Llama-3.1-70B-Instruct (AWQ-INT4), run locally with vLLM.
• Sampling: 40 train / 50 test examples per latent, example context length 32, 10 activation quantiles.
• Scorers: detection and fuzzing. 95% confidence intervals are bootstrapped by resampling latents (2,000 resamples).

Scores are computed over a representative latent sample that includes rare features. Balanced accuracy sits near chance for the rarer latents, which are much harder to explain than high-frequency latents; frequency-weighted F1 upweights frequently-firing latents.

Companion variant: EleutherAI/sae-SmolLM2-1.7B-layer17-32x-embedskip. Both SAEs are evaluated on an identical token stream, so the comparison reflects the SAE, not sampling noise.