Downloads · 30 days
1
3% of all-time downloads
cayley-10b-k16-3l-mlp_in
markhenry/cayley-10b-k16-3l-mlp_in
cayley-10b-k16-3l-mlp_in is a machine learning model from markhenry. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Trained by @amack (aemack-org). Originally uploaded as aemack-org/cayley-10b (commit 80734cf58094f8b888e732abf405bd100e1476d2, since replaced with a different architecture).
All-time downloads
32
Public
Repo size
1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt1 GB · 100%
At a glance
- Access
- Public
- Created
- Mar 30, 2026
- Updated
- Mar 30, 2026
- SHA
- 069c567d
Model card
From the Hugging Face model README
cayley-10b-k16-3l-mlp_in
Trained by @amack (aemack-org). Originally uploaded as aemack-org/cayley-10b
(commit 80734cf58094f8b888e732abf405bd100e1476d2, since replaced with a different architecture).
Reaches 3.173 CE val loss after 16k iters (~10.5B tokens on FineWeb-Edu-10B).
Architecture
- 205M-param GPT (12 layers, 8 heads, d=1024)
- CayleySAE inserted at
mlp_inin every block (3-level hierarchy)
CayleySAE config
| Level | n | k | delta | Features |
|---|---|---|---|---|
| L0 | 10 | 16 | 0 (root) | 1,024 |
| L1 | 13 | 32 | 64 | 8,192 |
| L2 | 16 | 64 | 64 | 65,536 |
- Total features per layer: 74,752 (73x overcomplete)
- Active per token: 112
per_parent_budget=True,score_standardize=True- Location:
mlp_in(RMSNorm → CayleySAE → MLP in each block)
Training config
optimizer: muon (lr=0.006), adamw (lr=0.006)
lr_schedule: linear_warmdown, warmdown_frac=0.2
batch_size: 80, seq_len: 1024, gradient_accumulation_steps: 1
max_iters: 16000
- Created
- Mar 30, 2026
- Updated
- Mar 30, 2026