gemma-loupe

gemma-loupe

gemma-loupe is a complete set of sparse autoencoders for Google's Gemma 4 E2B model, covering all 35 text decoder layers. trained with sparsify.

what's in this repo

• 35 SAEs, one for each of the 35 decoder layers (see sae-gemma-4-E2B-32x-1B/language_model.layers.*/)
• max-activating examples for every feature across all layers (see sae-gemma-4-E2B-32x-1B/max_activations/)
• a getting started notebook showing how to load, encode, and browse features
• a finding a feature notebook showing a possible workflow for identifying features relevant to a concept of your choice

training details

max-activating examples

we sampled 10M tokens from RedPajama v2, storing the top 20 examples per feature. these are stored as parquet files with the following columns:

• feature: index (int)
• activation: peak activation value (float)
• token: the activating token (string)
• context: surrounding context window (string)
• context_tokens: individual tokens in the context window (JSON)
• token_activations: per-token activation values across the context (JSON)

quick start

from sparsify import SparseCoder

# load a single layer's SAE
sae = SparseCoder.load_from_hub(
    "rhizomatous/gemma-loupe",
    hookpoint="sae-gemma-4-E2B-32x-1B/language_model.layers.17",
    device="cuda",  # or "mps" / "cpu"
)

# load all 35 layers at once
saes = SparseCoder.load_many(
    "rhizomatous/gemma-loupe",
    device="cuda", # or "mps" / "cpu"
    pattern="sae-gemma-4-E2B-32x-1B/language_model.layers.*",
)

see the example notebook for a walkthrough including encoding text, inspecting per-token features, and browsing max-activating examples.

limitations

• text-only. Gemma 4 E2B is a trimodal AVLN, but these SAEs cover only the 35 text decoder layers. the vision layers and audio layers are not addressed. if you feel compelled, you could whack in an image or sound file and see what happens, but caveat utilitor!
• English-only. gemma-loupe was trained on English web text from RedPajama v2. feature quality on other languages or domains may vary.

architecture reference

for details on Gemma 4 E2B's architecture, see Maarten Grootendorst's excellent A Visual Guide to Gemma 4.

citation

if you use these SAEs in your work, please cite:

@misc{gemma-loupe,
    title={gemma-loupe: Sparse Autoencoders for Gemma 4 E2B},
    author={viv shaw},
    year={2026},
    url={https://huggingface.co/rhizomatous/gemma-loupe},
}