Downloads · 30 days
0
topk.tblock.gpt2.k32.x32
davidquarel/topk.tblock.gpt2.k32.x32
topk.tblock.gpt2.k32.x32 is a machine learning model from davidquarel. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
TopK SAE trained bewteen each layer for GPT-2 Trained on dataset "HuggingFaceFW/fineweb-edu" K=32 Expansion factor = 32 (768 32 = 24576)
Access
Public
Updated Jun 4, 2025
Parameters
124M
2.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.5 GB · 100%
At a glance
- Access
- Public
- Created
- Jun 4, 2025
- Updated
- Jun 4, 2025
- SHA
- 9fb1deda
Model card
From the Hugging Face model README
TopK SAE trained bewteen each layer for GPT-2
- Trained on dataset "HuggingFaceFW/fineweb-edu"
- K=32
- Expansion factor = 32 (768 * 32 = 24576)
gpt2_defaults = {
"data_dir": "data/fineweb_edu_10b",
"eval_interval": 250,
"eval_steps": 100,
"batch_size": 1,
"gradient_accumulation_steps": 32 // 1,
"learning_rate": 5e-4,
"warmup_steps": 750,
"max_steps": 7500,
"decay_lr": True,
"min_lr": 1e-4,
}
SAEConfig("topk.tblock.gpt2",
gpt_config = gpt_options['gpt2'],
n_features=tuple(768 * n for n in (32,)*13),
sae_variant=SAEVariant.TOPK,
top_k = (32,) * 13,
sae_keys=gen_sae_keys(n_features=13, loc="standard"),
),
- Created
- Jun 4, 2025
- Updated
- Jun 4, 2025