Downloads · 30 days
47
4% of all-time downloads
siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860
2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860 is a machine learning model from siddharthmb. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's feat…
Downloads · 30 days
47
4% of all-time downloads
All-time downloads
1.3K
Public
Parameters
3.6B
7.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors7.2 GB · 100%
From the Hugging Face model README
Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper).
Load it with the repo's model classes; the adapter branch is part of the checkpoint, and set_transcoders_enabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings):
from models.auto import load_tokenizer, load_transcoder_model
model = load_transcoder_model("siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860", device_map="auto")
tokenizer = load_tokenizer("siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860")
Chat with it, or steer one feature by its cantor id (--steer CANTOR_ID:STRENGTH):
uv run python -m analysis.chat.cli siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860 --prompt "Hi"
Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.build_graphs --adapter_checkpoint siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860 ... (--help lists the prompt, GemmaScope and budget flags).
Produced by python -m training.train from transcoder-adapters at commit 6e94ae694a62 (Slurm job 14793860, on jagupard37.stanford.edu, started 2026-03-11T04:08:34.855260Z).
Card generated after the fact on 2026-10-11 from the W&B run's logged config, metadata and summary and the training log logs/train/14793860_20260310_205049.out; the run itself recorded no manifest. The training-config.yaml uploaded with the weights is this run's config (agrees with the W&B config on every key); train_config.yaml is the W&B config under today's file name, with the trainer options added after this run pinned to the behaviour of the time (--fp32_transcoder_params false, --bucket_by_length false, --attn_implementation sdpa, bridging.nmse_pooled: true).
Exact command (from the training log):
uv run python -m training.train --config training/configs/gemma2_2b.yaml training/configs/gemma-matrix/chat_truncate.yaml
logs/train/14793860_20260310_205049.outRetrain from the uploaded config: uv run python -m training.train --config train_config.yaml (its output_dir and wandb_run_name are this run's; override them for a new run).
google/gemma-2-2bgoogle/gemma-2-2b-itgoogle/gemma-2-2b-itmodel-00001-of-00002.safetensors, model-00002-of-00002.safetensors (index model.safetensors.index.json)config.json, generation_config.json, training-config.yamlFinal training step (from the W&B summary; this run computed no held-out token metrics): bridging/total is the summed objective, bridging/adapt_only the end-to-end KL of the adapter model to the reference, bridging/bridge the bridging KL of the mixed forward passes, bridging/nmse the layer-output NMSE, train/sparsity the mean active features per token (L0) and adapter_stats_avg/dead_features_1000 the mean count of features silent over the last 1000 steps.
/nlp/scr/siddharth/sparse-adaptation/checkpoints/gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793860_2026-03-10_2050_14793860 (not retained; the weights are in this repo)logs/train/14793860_20260310_205049.out