Downloads · 30 days
0
Silen/cosine-scored-saes-qwen3-8b
cosine-scored-saes-qwen3-8b is a machine learning model from Silen. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as mit.
SAE checkpoints for "Size Doesn't Matter: Cosine-Scored Sparse Autoencoders" (spotlight, ICML 2026 Workshop on Mechanistic Interpretability).
Downloads · 30 days
0
Access
Public
Updated Jun 22, 2026
Repo size
6.4 GB
Likes
0
Public
Click a slice to open those files.
.pt6.4 GB · 100%
From the Hugging Face model README
SAE checkpoints for "Size Doesn't Matter: Cosine-Scored Sparse Autoencoders" (spotlight, ICML 2026 Workshop on Mechanistic Interpretability).
Code: https://github.com/SilenNaihin/cosine-scored-saes
These are the 500M-token headline SAEs on Qwen3-8B, layer 18 residual stream
(blocks.18.hook_resid_post), d_sae = 65,536, BatchTopK k = 80, trained on
FineWeb with the SAEBench/saprmarks recipe (Adam lr 5e-5, AuxK dead-feature loss
alpha=1/32, unit-norm decoder, geometric-median b_dec init).
The cosine encoder replaces the standard inner-product pre-activation
s_i(x) = <w_i, x_c> + b_i with a learned norm-scaled cosine score
s_i(x) = exp(b) * ||x_c||^a * cos(x_c, w_i) + b_enc,i, where a interpolates
between pure cosine (a=0) and inner product (a=1). Encoder rows are unit-normalized.
| Folder | Variant | a | Top-1 probing |
|---|---|---|---|
standard/ | Standard BatchTopK (inner product) | — | 0.667 |
global-a/ | Adaptive Cosine (single shared a) | 0.258 | 0.808 |
perfeature/ | Per-Feature Adaptive Cosine (a_i = a_base + delta_i) | mean 0.076 | 0.813 |
Mean +/- SD over seeds {42, 123, 456}. FVE = fraction of variance explained; top-1 = SAEBench single-feature sparse-probing top-1 accuracy (8 datasets).
| Variant | FVE | Probing top-1 | Learned a |
|---|---|---|---|
| Standard | 0.7702 +/- 0.0002 | 0.6669 +/- 0.0026 | — |
Global a | 0.7690 +/- 0.0000 | 0.8081 +/- 0.0059 | 0.2577 +/- 0.0007 |
| Per-feature | 0.7707 +/- 0.0002 | 0.8128 +/- 0.0126 | 0.0759 +/- 0.0001 |
At matched reconstruction (FVE within 0.4%), the cosine variants improve single-feature sparse-probing top-1 by +14.1% (global) and +14.6% (per-feature). The learned norm-dependence exponent stays far below the inner-product limit (a=1) across all seeds. The checkpoint published per variant is the best of seeds 123/456 by top-1.
Each *_final.pt is a dict: {"state_dict", "num_tokens_since_fired", "step"}.
state_dict keys: W_enc [65536,4096], b_enc [65536], W_dec [65536,4096],
b_dec [4096], plus scale_a, scale_b, threshold for the cosine variants.
SAE class definitions are in the code repo
(experiments/40_saprmarks_recipe/exp40_karvonen_recipe.py).
@inproceedings{naihin2026cosine,
title={Size Doesn't Matter: Cosine-Scored Sparse Autoencoders},
author={Naihin, Silen and Stambler, Lev},
booktitle={ICML 2026 Workshop on Mechanistic Interpretability},
year={2026}
}