Downloads · 30 days
6
14% of all-time downloads
cayley-32k-2L-mlp_in-v2
markhenry/cayley-32k-2L-mlp_in-v2
cayley-32k-2L-mlp_in-v2 is a machine learning model from markhenry. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Successor to markhenry/cayley-32k-2L-mlpin that enables cayleyforwardstandardized=True (the "zombie-features fix": forwards z-scored CayleySAE scores into the reconstruction). Same backbone, same Cayley hierarchy.
All-time downloads
43
Public
Repo size
986 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt986 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Apr 29, 2026
- Updated
- Apr 29, 2026
- SHA
- be2057d1
Model card
From the Hugging Face model README
cayley-32k-2L-mlp_in-v2
Successor to markhenry/cayley-32k-2L-mlp_in
that enables cayley_forward_standardized=True (the "zombie-features fix":
forwards z-scored CayleySAE scores into the reconstruction). Same backbone,
same Cayley hierarchy.
Architecture
| Parameter | Value |
|---|---|
| n_layer | 12 |
| n_head | 8 |
| n_embd | 1024 |
| block_size | 1024 |
| vocab_size | 50304 |
| bias | False |
| norm | RMSNorm (affine) |
| MLP | GELU, 4x expansion |
| tokenizer | GPT-2 (tiktoken) |
| dtype | bfloat16 |
| sparsity_mode | cayley |
| cayley_locations | mlp_in |
| cayley_levels | 10,16,0; 15,32,256 |
| cayley_per_parent_budget | True |
| cayley_score_standardize | True |
| cayley_forward_standardized | True ← changed from v1 |
L0 = (1024 features, k=16) → L1 = (32768 features, k=32, delta=256).
Training
| Parameter | Value |
|---|---|
| optimizer | Muon (hidden 2D) + AdamW (embeddings) |
| muon_lr | 0.012 |
| muon_min_lr | 1.5e-04 |
| adamw_lr | 0.012 |
| adamw_min_lr | 1.5e-04 |
| lr_schedule | linear_warmdown (warmdown_frac=0.9) |
| batch_size | 24 |
| seq_len | 1024 |
| grad_accum_steps | 56 |
| max_iters | 7267 |
| tokens seen | ~10.0B |
| dataset | FineWeb-Edu-10B |
| best_val_loss | 3.2155 |
Comparison vs v1
v2 takes a small CE hit (3.2155 vs 3.1162) in exchange for a measurably cleaner Cayley hierarchy at layer 6. Headline hierarchy metrics (parent_level=0, ~492k val tokens):
| metric | v1 | v2 |
|---|---|---|
| effective_parent_count / frac_children_in_healthy_band | 0.060 | 0.155 |
| coactivation / frac_parents_saturated_children | 0.093 | 0.022 |
| sibling_mi / global_weighted_mean_npmi | 0.236 | 0.202 |
| parent_attribution / mean_attribution_quality | 1.78 | 1.49 |
cayley_forward_standardized removes most of v1's saturated/decorative
parents (−76 %) and triples the share of children in the healthy 2–5
effective-parent band. Sibling exclusivity (lower NPMI) improves slightly.
The single regression is parent_attribution, which fell along with v1's
parent over-firing.
See mh/reports/29-zombie-features-forward-standardized.md in the
sparse-nanogpt-private
repo for the broader experiment context.
- License
- mit
- Created
- Apr 29, 2026
- Updated
- Apr 29, 2026