gemma-4-saes

Gemma-4 SAEs for use with the SAELens library

This repository contains the following SAEs:

Gemma-4-E2B

• gemma-4-e2b/btk-mat-layer-6-k-100
• gemma-4-e2b/btk-mat-layer-17-k-100
• gemma-4-e2b/btk-mat-layer-28-k-100

Gemma-4-E4B

• gemma-4-e4b/btk-mat-layer-7-k-100
• gemma-4-e4b/btk-mat-layer-21-k-100
• gemma-4-e4b/btk-mat-layer-35-k-100

Gemma-4-31B

• gemma-4-31b/btk-mat-layer-30-k-100

Load these SAEs using SAELens as below:

from sae_lens import SAE

sae = SAE.from_pretrained("decoderesearch/gemma-4-saes", "<sae_id>")

About these SAEs

These SAEs are all Matryoshka BatchTopK SAEs trained on the Pile Uncopyrighted using SAELens.

The Gemma-4-E2B and Gemma-4-E4B SAEs each have width 65k latents, and 2 inner Matryoshka prefixes of 2k latents and 16k latents. The Gemma-4-31B SAE has width 131k latents and 2 inner Matryoshka prefixes of 4k latents and 32k latents.

Reproducing training

For complete details of how these SAEs were trained, refer to the runner_cfg.json in each SAE directory. This config includes all hyperparameters passed to the SAELens trainer when training the SAEs.

Citation

@misc{decode2026gemma4saes,
  author       = {Chanin, David and Lin, Johnny},
  title        = {{Gemma-4} Sparse Autoencoders},
  year         = {2026},
  organization = {Decode Research},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/decoderesearch/gemma-4-saes}}
}

Acknowledgements

These SAEs were trained thanks to compute provided by Modal.