sasa-gemma3-runs

SASA layer-17 pilot sweep for Gemma 3 1B (300M tokens)

The 15-arm hyperparameter sweep that selected the center configuration used by the full-depth companion repo (sasa-gemma3-layers). All arms are subspace-aware sparse autoencoders (SASA; Dalili & Mahdavi 2026, arXiv:2606.06333) trained on blocks.17.hook_resid_post of google/gemma-3-1b-it from one shared, deterministic activation cache (309.7M tokens, FineWeb-Edu, fp32, seed 42), so every cross-arm comparison holds the token stream fixed.

Arms

Tier A — rank × sparsity: r ∈ {3, 6, 12} × ℓ0 ∈ {30, 60, 120} at m = 18,432. Tier B — width and nuclear penalty: m = 65,532 at ℓ0 = 60; nuclear λ ∈ {0, 10} ablations of the center (center itself is λ = 100). Tier C — seed control (seed 43). Tier D — learning rate {1e-4, 4e-4} and composition-target nuclear penalty.

Headline held-out results (FineWeb-Edu CC-MAIN-2025-08, 256 × 512 tokens)

• Seed noise floor: 0.0016 FVU (seed 42 vs 43) — gaps above it are real.
• Nuclear penalty is a free health regularizer: FVU flat across λ ∈ {0,10,100} (0.158–0.161), dead groups 36–38% → 22%. It does not act through rank: the per-step decoder row renorm pins every block's spectrum at ≈98% of the isotropic maximum at every λ. Do not cite these weights as evidence of adaptive rank control.
• Rank trades FVU against dictionary health at this layer: smaller r wins FVU, r = 12 keeps the most features alive.

Repository layout

runs/sweep300/<arm>/     # weights + cfg + full loss logs (300M-token runs)
runs/sweep/<arm>/        # earlier short runs (superseded; kept for provenance)
results/heldout300_*.json  # held-out eval per arm (each file also carries the
                           # Gemma Scope 2 control row for its comparison)
results/heldout_primary.json

Sweep-trained exports carry empty model/hook metadata; pass --model google/gemma-3-1b-it --hook-name blocks.17.hook_resid_post explicitly when evaluating. The activation cache is not included but is fully reproducible: the token stream is determined by (preset, seed, shard split) in the code release's kvds/cache.py.

Provenance and caution

Trained August 2026 on an 8×A100 node. Weights and results are original artifacts of this project (license above); the SASA training code is vendored from github.com/arshandalili/sasa, which at time of writing carries no license — consult that repository before reusing the code. Gemma Terms of Use apply to the base model.

Citation: paper reference to be added upon publication.