Downloads · 30 days
0
musicakamusic/crimsonred-sae
crimsonred-sae is a machine learning model from musicakamusic. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
TopK Sparse Autoencoders trained on Gemma 4 residual-stream activations, part of the CrimsonRed emotion-vector project.
Downloads · 30 days
0
Access
Public
Updated Jul 22, 2026
Repo size
1.2 GB
Likes
0
Public
Click a slice to open those files.
.pt1.2 GB · 100%
From the Hugging Face model README
TopK Sparse Autoencoders trained on Gemma 4 residual-stream activations, part of the CrimsonRed emotion-vector project.
To our knowledge these are the first general-purpose SAEs published for Gemma 4 E2B / E4B. (Existing public SAEs cover Gemma 2 (Gemma Scope), Gemma 4 31B, and the Gemma 4 26B MoE — not the on-device E2B/E4B dense models.)
| Model | Layer | d_model | d_sae | TopK | Recon cosine | SAE weights | Emotion vectors |
|---|---|---|---|---|---|---|---|
| Gemma 4 E2B (IT) | 17 | 1536 | 24576 (16×) | 50 | 0.982 | gemma_e2b/layer17.sae.pt | gemma_e2b/emotion_vectors.pt |
| Gemma 4 E4B (IT) | 21 | 2560 | 40960 (16×) | 50 | 0.995 | gemma_e4b/layer21.sae.pt | gemma_e4b/emotion_vectors.pt |
Each emotion_vectors.pt is {layer: {emotion_name: float32[d_sae]}} — 171 contrastive
emotion profiles (emotion SAE profile − neutral SAE profile), drop-in for
crimsonred.sae_scoring.SAEScoringStrategy.
Residualizes a held-out query against the neutral profile (matching how the contrastive vectors are built), then scores it against all 171 stored emotion vectors by cosine. 171-way classification; random chance is 0.6%.
| Model | cos top-1 | cos top-5 |
|---|---|---|
| Qwen (reference) | 0.901 | 1.000 |
| Gemma E2B | 0.865 | 1.000 |
| Gemma E4B | 0.702 | 0.971 |
E4B's lower accuracy reflects genuine near-synonym confusions (energized→excited, terrified→uneasy), not degenerate collapse — its misses are all within-family.
encode: pre = x @ W_enc.T + b_enc
acts = relu(pre)
f = topk(acts, k=50) # keep 50 largest, zero the rest
decode: x_hat = f @ W_dec.T + b_dec
Decoder columns are kept at unit norm during training.
Each layer{N}.sae.pt is a torch.load(..., weights_only=True) dict:
| Key | Shape | dtype |
|---|---|---|
W_enc | [d_sae, d_model] | float32 |
W_dec | [d_model, d_sae] | float32 |
b_enc | [d_sae] | float32 |
b_dec | [d_model] | float32 |
This format is drop-in compatible with crimsonred.sae_scoring.load_sae_weights.
| Metric | E2B (L17) | E4B (L21) |
|---|---|---|
| mean cosine | 0.9822 | 0.9953 |
| median cosine | 0.9830 | 0.9955 |
| MSE | 0.1212 | 0.0257 |
| variance explained | 0.4052 | 0.2676 |
| mean L0 | 50.0 (exact) | 50.0 (exact) |
| feature utilization | 0.88% | 0.12% |
Feature utilization is low because the SAEs were trained on emotion-domain activations, which occupy a lower-dimensional manifold than general text — most general-purpose features never fire. The reconstruction cosines (0.982 / 0.995) confirm both SAEs capture the emotion-relevant subspace well.
import torch
from crimsonred.sae_scoring import SAEScoringStrategy, load_sae_weights
# Load pre-computed emotion vectors (fastest path — no activation files needed)
raw = torch.load("gemma_e2b/emotion_vectors.pt", weights_only=False)
vectors = raw[17] # {emotion_name: float32[24576]}
# Score new hidden states
strategy = SAEScoringStrategy(vectors, sae_dir="gemma_e2b", layer=17)
scores = strategy.score_emotions(hidden_states, layer=17) # list of EmotionScore
To recompute emotion vectors from activations:
from crimsonred.sae_scoring import compute_sae_emotion_vectors
vectors = compute_sae_emotion_vectors(
activations_dir="data/activations_gemma_e2b",
sae_dir="gemma_e2b",
layer=17,
neutral_path="data/neutral_activations_gemma_e2b.pt",
k=50,
)
@misc{crimsonred_sae_2026,
title = {CrimsonRed SAE: TopK Sparse Autoencoders for Gemma 4 E2B/E4B},
author = {CrimsonRed Project},
year = {2026},
howpublished = {Hugging Face},
url = {https://huggingface.co/musicakamusic/crimsonred-sae}
}