sae-SmolLM2-1.7B-layer17-32x-embedskip

Skip-SAE for SmolLM2-1.7B, layer 17

A TopK sparse autoencoder trained with sparsify on the residual stream of HuggingFaceTB/SmolLM2-1.7B at layer 17 (0-indexed, out of 24 layers — roughly 3/4 of the way through the model), with an added learned affine skip connection from the model's embedding activations to the SAE output (as in a skip transcoder), rather than from this hookpoint's own input. This is meant to control for aspects of the reconstruction explainable by generic token/dataset geometry already present at the embedding layer, so the remaining latent features are hopefully more semantically interesting.

Training details

• Dataset: EleutherAI/SmolLM2-135M-10B, one full epoch (19,044 steps)
• Expansion factor: 32x (65,536 latents, d_in=2048)
• k: 32
• Optimizer: Adam (see train_config.json for the full training config)
• Final FVU: 0.00902, dead latents: 0%

See train_config.json for the complete sparsify.TrainConfig used.

Usage

from sparsify import Sae

sae = Sae.load_from_hub("EleutherAI/sae-SmolLM2-1.7B-layer17-32x-embedskip")
# sae.W_embed_skip, sae.b_embed_skip hold the learned embedding-activation skip connection

Automated interpretability

Explanations were generated and scored with delphi automated interpretability over a growing sample of latents (currently the first 900 latent indices, of which 899 cleared the ≥200 activating-example threshold and were scored).

Protocol

• Activations cached over 200M tokens of the training distribution (EleutherAI/SmolLM2-135M-10B, BOS-filtered). Whole-SAE at this budget: 0 dead latents, and 99.8% of latents fire ≥200× (65,391 / 65,536).
• Explainer & scorer: Llama-3.1-70B-Instruct (AWQ-INT4), run locally with vLLM.
• Sampling: 40 train / 50 test examples per latent, example context length 32, 10 activation quantiles.
• Scorers: detection and fuzzing. 95% confidence intervals are bootstrapped by resampling latents (2,000 resamples).

Scores are computed over a representative latent sample that includes rare features. Balanced accuracy sits near chance for the rarer latents, which are much harder to explain than high-frequency latents; frequency-weighted F1 upweights frequently-firing latents.

Companion variant: EleutherAI/sae-SmolLM2-1.7B-layer17-32x. Both SAEs are evaluated on an identical token stream, so the comparison reflects the SAE, not sampling noise.