Downloads · 30 days
4
7% of all-time downloads
cayley-131k-3L-mlp_in-v3
markhenry/cayley-131k-3L-mlp_in-v3
cayley-131k-3L-mlp_in-v3 is a machine learning model from markhenry. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Successor to markhenry/cayley-131k-3L-mlpin-v2 that enables cayleyforwardstandardized=True (the "zombie-features fix": forwards z-scored CayleySAE scores into the reconstruction). Same backbone, same 3-level Cayley hi…
All-time downloads
61
Public
Repo size
2.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt2.7 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- May 1, 2026
- Updated
- May 1, 2026
- SHA
- 3282536a
Model card
From the Hugging Face model README
cayley-131k-3L-mlp_in-v3
Successor to markhenry/cayley-131k-3L-mlp_in-v2
that enables cayley_forward_standardized=True (the "zombie-features fix":
forwards z-scored CayleySAE scores into the reconstruction). Same backbone,
same 3-level Cayley hierarchy.
Architecture
| Parameter | Value |
|---|---|
| n_layer | 12 |
| n_head | 8 |
| n_embd | 1024 |
| block_size | 1024 |
| vocab_size | 50304 |
| bias | False |
| norm | RMSNorm (affine) |
| MLP | GELU, 4x expansion |
| pos_encoding | RoPE (theta=10000) |
| tokenizer | GPT-2 (tiktoken) |
| dtype | bfloat16 |
| sparsity_mode | cayley |
| cayley_locations | mlp_in |
| cayley_levels | 10,16,0; 15,32,256; 17,64,256 |
| cayley_per_parent_budget | True |
| cayley_score_standardize | True |
| cayley_forward_standardized | True ← changed from v2 |
L0 = (1024 features, k=16) → L1 = (32768 features, k=32, delta=256) → L2 = (131072 features, k=64, delta=256).
Training
| Parameter | Value |
|---|---|
| optimizer | Muon (hidden 2D) + AdamW (embeddings) |
| muon_lr | 0.012 |
| muon_min_lr | 1.5e-04 |
| adamw_lr | 0.012 |
| adamw_min_lr | 1.5e-04 |
| lr_schedule | linear_warmdown (warmdown_frac=0.9) |
| warmup_iters | 200 |
| batch_size | 32 |
| seq_len | 1024 |
| grad_accum_steps | 48 |
| max_iters | 10173 |
| tokens seen | ~16.0B |
| dataset | FineWeb-Edu-100B |
| best_val_loss | 3.138 |
Comparison vs v2
v3 takes a small CE hit (3.138 vs v2's 3.131 at 21B tokens) in exchange for
the zombie-features fix. The qualitative pattern from
mh/reports/29-zombie-features-forward-standardized.md (cleaner Cayley
hierarchy, fewer saturated parents, more children in the healthy
effective-parent band) carried over for the 12L 32k 2L sibling
(markhenry/cayley-32k-2L-mlp_in-v2);
production validation on the 131k 3L variant pending.
Purpose
Primary interpretability subject model for 131k-3L CayleySAE experiments with the post-zombie-fix codebase. Successor in the same lineage as markhenry/cayley-131k-3L-mlp_in-v2 and aemack-org/cayley-10b, used for feature-profiling, circuit-discovery, and alignment-tax comparisons against vanilla baselines.
- License
- mit
- Created
- May 1, 2026
- Updated
- May 1, 2026