Downloads · 30 days
18
22% of all-time downloads
siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr10000_lr8e-04_bs2_sl14700755
2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr10000_lr8e-04_bs2_sl14700755 is a machine learning model from siddharthmb. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's feat…
Downloads · 30 days
18
22% of all-time downloads
All-time downloads
82
Public
Parameters
3.6B
7.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors14.4 GB · 100%
From the Hugging Face model README
Sparse transcoder adapter for google/gemma-2-2b: a ReLU branch dec(relu(enc(x))) on every MLP of the frozen base model, trained in bridging mode so that base plus branch matches google/gemma-2-2b-it. The branch's features are the fine-tuning, made sparse and inspectable (paper).
Load it with the repo's model classes; the adapter branch is part of the checkpoint, and set_transcoders_enabled(False) gives the hybrid model (base MLPs with the reference model's attention and embeddings):
from models.auto import load_tokenizer, load_transcoder_model
model = load_transcoder_model("siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr10000_lr8e-04_bs2_sl14700755", device_map="auto")
tokenizer = load_tokenizer("siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr10000_lr8e-04_bs2_sl14700755")
Chat with it, or steer one feature by its cantor id (--steer CANTOR_ID:STRENGTH):
uv run python -m analysis.chat.cli siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr10000_lr8e-04_bs2_sl14700755 --prompt "Hi"
Attribution graphs of this adapter: uv run --extra viz python -m analysis.attribution.graphs.build_graphs --adapter_checkpoint siddharthmb/2026.TA.gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr10000_lr8e-04_bs2_sl14700755 ... (--help lists the prompt, GemmaScope and budget flags).
Produced by python -m training.train from transcoder-adapters at commit 1d6acc758c22 (Slurm job 14700755, on jagupard33.stanford.edu, started 2026-03-03T07:17:23.136020Z).
Card generated after the fact on 2026-10-11 from the W&B run's logged config, metadata and summary and the training log logs/train/14700755.out; the run itself recorded no manifest. The training log of the time did not print the invocation; the command is the argv the W&B run recorded. The run predates the datasets list: its config named one mixed source (FINEWEB_LMYSYSCHAT_MIXED: science-of-finetuning/fineweb-1m-sample and lmsys/lmsys-chat-1m, 1000 tokens each, length policy truncate) and dataset_rows: 10000, a uniform sample of 10000 of the 1900000 concatenated rows; train_config.yaml writes that as two entries weighted by the row counts the log reports (900000 and 1000000) with total_rows: 10000 and weight_by: rows, which matches the sample in expectation, not row for row. The weights were uploaded without a training config and the checkpoint dir was not retained; the train_config.yaml beside this card is the config the run logged to W&B, in today's schema, with the trainer options added after this run pinned to the behaviour of the time.
Exact command (from the W&B run's recorded argv):
python -m training.train --config training/configs/gemma2_2b.yaml --batch_size=2
logs/train/14700755.outRetrain from the uploaded config: uv run python -m training.train --config train_config.yaml (its output_dir and wandb_run_name are this run's; override them for a new run).
google/gemma-2-2bgoogle/gemma-2-2b-itgoogle/gemma-2-2b-itmodel-00001-of-00002.safetensors, model-00002-of-00002.safetensors (index model.safetensors.index.json)chat_template.jinja, special_tokens_map.json, tokenizer.json, tokenizer_config.jsonconfig.json, generation_config.jsonlatest_step_5000/ (the run's last periodic save, same layout)Final training step (from the W&B summary; this run computed no held-out token metrics): bridging/total is the summed objective, bridging/adapt_only the end-to-end KL of the adapter model to the reference, bridging/bridge the bridging KL of the mixed forward passes, bridging/nmse the layer-output NMSE, train/sparsity the mean active features per token (L0) and adapter_stats_avg/dead_features_1000 the mean count of features silent over the last 1000 steps.
/nlp/scr/siddharth/sparse-adaptation/checkpoints/gemma2_2b_tc8192_decb_l1w0.001_tarbb_lb2.0_ln1_dr10000_lr8e-04_bs2_sl14700755_2026-03-02_2314_14700755 (not retained; the weights are in this repo)logs/train/14700755.out