Downloads · 30 days
0
gemma-4-e2b-scope-v1-L17-batchtopk-k64-seed17
Solshine/gemma-4-e2b-scope-v1-L17-batchtopk-k64-seed17
gemma-4-e2b-scope-v1-L17-batchtopk-k64-seed17 is a machine learning model from Solshine. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
BatchTopK Sparse Autoencoder trained on residual-stream activations from Gemma 4 E2B at layer 17 (relative depth ≈ 49 %), on FineWeb-Edu (pretraining-distribution) text under bitsandbytes 4-bit NF4 quantization.
Access
Public
Updated Apr 30, 2026
Repo size
7.3 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.pt906 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026
- SHA
- 0ceb25ea
Base models
Datasets
Model card
From the Hugging Face model README
gemma-4-e2b-scope-v1-L17-batchtopk-k64-seed17
BatchTopK Sparse Autoencoder trained on residual-stream activations from Gemma 4 E2B at layer 17 (relative depth ≈ 49 %), on FineWeb-Edu (pretraining-distribution) text under bitsandbytes 4-bit NF4 quantization.
Training progress — Checkpoint 2 (2026-04-30)
| Field | Value |
|---|---|
| Tokens seen | 16,001,024 |
| Target total | 100,000,000 |
| Progress | 16.0 % |
| Training steps | 15,626 |
| Last checkpoint | 2026-04-30T10:38:59 UTC |
Training metrics (Checkpoint 2)
| Metric | Checkpoint 1 (~8M tok) | Checkpoint 2 (16M tok) |
|---|---|---|
| Loss | ~0.654 | 0.586 |
| Explained variance | ~0.770 | 0.831 |
| Peak EV (Ckpt 2) | — | 0.849 @ step ~15,350 |
| L0 | 64 | 64 |
| Alive features (frac) | ~62 % | ~62 % |
Training is ongoing — weights update with each checkpoint push.
Hyperparameters
| Architecture | BatchTopK (Bussmann et al. arXiv:2412.06410) |
| d_in | 1536 |
| d_sae | 24576 (16× expansion) |
| k | 64 |
| Seed | 17 |
| Layer | 17 |
| Base model | google/gemma-4-E2B |
| Quantization | bitsandbytes NF4, fp16 compute |
| Optimizer | Adam, lr=3e-4 |
| Batch size | 1024 activations |
| Dataset | HuggingFaceFW/fineweb-edu (sample-10BT), streaming, seed=17 |
| Aux-k coefficient | 0.0625 |
| Decoder norm | Unit-norm per Gemma Scope recipe |
Usage
Load weights with SAELens-compatible state-dict keys:
import torch, json
from huggingface_hub import hf_hub_download
repo = "Solshine/gemma-4-e2b-scope-v1-L17-batchtopk-k64-seed17"
cfg = json.loads(open(hf_hub_download(repo, "cfg.json")).read())
state = torch.load(hf_hub_download(repo, "sae_weights.pt"), map_location="cpu", weights_only=True)
# Keys: W_enc [d_in, d_sae], W_dec [d_sae, d_in], b_enc [d_sae], b_dec [d_in]
Hook into Gemma 4 E2B at layer 17 to collect residual-stream activations, then encode with the SAE. Per-example TopK (not batch-level) for inference.
Research context
This SAE is part of an ongoing deception-interpretability research program examining whether behavioral distinctions (honest vs. deceptive model outputs) leave recoverable traces in SAE feature space. Training on the pretraining distribution (FineWeb-Edu) establishes a general-purpose feature vocabulary for Gemma 4 E2B; subsequent experiments probe this vocabulary against decision-incentive behavioral scenarios.
Live W&B run: https://wandb.ai/caleb-deleeuw/gemma4-sae-scope/runs/gemma-4-e2b-scope-v1-L17-batchtopk-k64-seed17
- License
- apache-2.0
- Languages
- en
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026