Downloads · 30 days
6
29% of all-time downloads
cayley-10b-k8-3l-mlp_in
markhenry/cayley-10b-k8-3l-mlp_in
cayley-10b-k8-3l-mlp_in is a machine learning model from markhenry. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
CayleySAE GPT trained with half-k topology: k=8/16/32 per level instead of the standard k=16/32/64.
All-time downloads
21
Public
Repo size
1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt1 GB · 100%
At a glance
- Access
- Public
- Created
- Mar 28, 2026
- Updated
- Mar 28, 2026
- SHA
- 1fddb5cd
Model card
From the Hugging Face model README
cayley-10b-halfk
CayleySAE GPT trained with half-k topology: k=8/16/32 per level instead of the standard k=16/32/64.
Architecture
- 12 layers, 8 heads, d=1024 (~205M params)
- CayleySAE at mlp_in: L0 (1024, k=8) → L1 (8192, k=16) → L2 (65536, k=32)
- Trained on FineWeb-Edu-10B for 16k iters
Results
- Best val loss: 3.1816
- Compare: cayley-10b (standard k) val loss 3.173
- Created
- Mar 28, 2026
- Updated
- Mar 28, 2026