Downloads · 30 days
0
111M-l12w720-owt-norm-gate-20260427
SubspaceTransformer/111M-l12w720-owt-norm-gate-20260427
111M-l12w720-owt-norm-gate-20260427 is a machine learning model from SubspaceTransformer. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Parameter-matched baseline and subspace transformer checkpoints.
Access
Public
Updated Apr 30, 2026
Repo size
2.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt2.7 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026
- SHA
- 7a5d8f97
Model card
From the Hugging Face model README
111M-l12w720-owt-norm-gate-20260427
Parameter-matched baseline and subspace transformer checkpoints.
Model Details
| Baseline | Subspace | |
|---|---|---|
| Parameters | 110,855,533 | 110,856,398 |
| Layers | 12 | 12 |
| Width | 720 | 720 |
| Heads | 12 | 12 |
| MLP Ratio | 4.0 | 5.2 |
| Q/K/V Fractions | - | 0.2/0.2/0.6 |
| Readout | - | v |
| Training Steps | 19,999 | 19,999 |
| Context Length | 1024 | 1024 |
| Vocab Size | 50,257 | 50,257 |
| Best Val Loss | 3.0083 | 3.0551 |
Files
baseline/model.pt- Baseline transformer checkpointsubspace/model.pt- Subspace transformer checkpointbaseline/config.json- Baseline resolved configsubspace/config.json- Subspace resolved config
- License
- apache-2.0
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026