Downloads · 30 days
13
25% of all-time downloads
siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862
2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862 is a machine learning model from siddharthmb. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's feat…
Downloads · 30 days
13
25% of all-time downloads
All-time downloads
51
Public
Parameters
3.6B
7.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors14.4 GB · 100%
From the Hugging Face model README
Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper).
Load it with the repo's model classes; the adapter branch is part of the checkpoint, and set_transcoders_enabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings):
from models.auto import load_tokenizer, load_transcoder_model
model = load_transcoder_model("siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862", device_map="auto")
tokenizer = load_tokenizer("siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862")
Chat with it, or steer one feature by its cantor id (--steer CANTOR_ID:STRENGTH):
uv run python -m analysis.chat.cli siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862 --prompt "Hi"
Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.build_graphs --adapter_checkpoint siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862 ... (--help lists the prompt, GemmaScope and budget flags).
Produced by python -m training.train from transcoder-adapters at commit 6e94ae694a62 (Slurm job 14793862, on jagupard35.stanford.edu, started 2026-03-11T03:58:58.870601Z).
Card generated after the fact on 2026-10-10 from the W&B run's logged config and summary and the training log logs/train/14793862_20260310_205055.out; the run itself recorded no manifest. The run's final push to the Hub crashed after the weights went up, so the repo carried no training config, and the checkpoint dir was not retained; the train_config.yaml beside this card is the config the run logged to W&B, with the trainer options added after this run pinned to the behaviour of the time.
Exact command (from the run manifest):
uv run python -m training.train --config training/configs/gemma2_2b.yaml training/configs/gemma-matrix/mixed_truncate.yaml
logs/train/14793862_20260310_205055.outRetrain from the uploaded config: uv run python -m training.train --config train_config.yaml (its output_dir and wandb_run_name are this run's; override them for a new run).
google/gemma-2-2bgoogle/gemma-2-2b-itgoogle/gemma-2-2b-itmodel-00001-of-00002.safetensors, model-00002-of-00002.safetensors (index model.safetensors.index.json)chat_template.jinja, special_tokens_map.json, tokenizer.json, tokenizer_config.jsonconfig.json, generation_config.jsonlatest_step_5000/ (the run's last periodic save, same layout)Final training step (from the W&B summary; this run computed no held-out token metrics): bridging/total is the summed objective, bridging/adapt_only the end-to-end KL of the adapter model to the reference, bridging/bridge the bridging KL of the mixed forward passes, bridging/nmse the layer-output NMSE, train/sparsity the mean active features per token (L0) and adapter_stats_avg/dead_features_1000 the mean count of features silent over the last 1000 steps.
/nlp/scr/siddharth/sparse-adaptation/checkpoints/gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr20000_lr8e-04_bs4_sl14793862_2026-03-10_2051_14793862 (the weights are in this repo)logs/train/14793862_20260310_205055.out