Downloads · 30 days
8
11% of all-time downloads
cayley-32k-2L-mlp_in
markhenry/cayley-32k-2L-mlp_in
cayley-32k-2L-mlp_in is a machine learning model from markhenry. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
GPT with a CayleySAE sparsity bottleneck inserted at mlpin in every block. Trained as a successor to aemack-org/cayley-10b with better val loss and the 32k-2L hierarchy.
All-time downloads
76
Public
Repo size
978 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt978 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Apr 9, 2026
- Updated
- Apr 9, 2026
- SHA
- aaaea81d
Model card
From the Hugging Face model README
cayley-32k-2L-mlp_in
GPT with a CayleySAE sparsity bottleneck inserted at mlp_in in every block.
Trained as a successor to aemack-org/cayley-10b
with better val loss and the 32k-2L hierarchy.
Architecture
| Parameter | Value |
|---|---|
| n_layer | 12 |
| n_head | 8 |
| n_embd | 1024 |
| block_size | 1024 |
| vocab_size | 50304 |
| bias | False |
| norm | RMSNorm (affine) |
| MLP | GELU, 4x expansion |
| tokenizer | GPT-2 (tiktoken) |
| dtype | bfloat16 |
| sparsity_mode | cayley |
| cayley_locations | mlp_in |
| cayley_levels | 10,16,0; 15,32,256 |
| cayley_per_parent_budget | True |
| cayley_score_standardize | True |
The cayley_levels entry [L, k, delta] per row defines a level with m = 2**L
features, selecting top-k per token (with per-parent budget delta at child
levels). For 10,16,0; 15,32,256 that is L0 = (1024 features, k=16) →
L1 = (32768 features, k=32, delta=256).
Training
| Parameter | Value |
|---|---|
| optimizer | Muon (hidden 2D) + AdamW (embeddings) |
| muon_lr | 0.001 |
| muon_min_lr | 5e-05 |
| adamw_lr | 0.001 |
| adamw_min_lr | 5e-05 |
| lr_schedule | linear_warmdown (warmdown_frac=0.7) |
| batch_size | 24 |
| seq_len | 1024 |
| grad_accum_steps | 64 |
| max_iters | 6358 |
| tokens seen | ~10.0B |
| dataset | FineWeb-Edu-100B (10B-token slice) |
| best_val_loss | 3.1162 |
Purpose
Successor run to aemack-org/cayley-10b. Same backbone, different CayleySAE
hierarchy (2L / 32k leaves vs. the original 3L / 65k), larger training budget,
and score standardization enabled. Target: beat the 3.173 CE val loss of
cayley-10b at an equal-or-larger token budget, to establish a stronger
interpretability subject model.
- License
- mit
- Created
- Apr 9, 2026
- Updated
- Apr 9, 2026