Downloads · 30 days
9
20% of all-time downloads
vanilla-v5-parity
markhenry/vanilla-v5-parity
vanilla-v5-parity is a machine learning model from markhenry. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Vanilla GPT baseline trained to compare against aemack-org/cayley-10b.
All-time downloads
44
Public
Repo size
920 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt920 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Mar 24, 2026
- Updated
- Mar 24, 2026
- SHA
- 270c828d
Model card
From the Hugging Face model README
vanilla-v5-parity
Vanilla GPT baseline trained to compare against aemack-org/cayley-10b.
Architecture
| Parameter | Value |
|---|---|
| n_layer | 12 |
| n_head | 8 |
| n_embd | 1024 |
| block_size | 1024 |
| vocab_size | 50304 |
| bias | False |
| norm | RMSNorm (affine) |
| MLP | GELU, 4x expansion |
| tokenizer | GPT-2 (tiktoken) |
| dtype | bfloat16 |
| sparsity | none (vanilla) |
Training
| Parameter | Value |
|---|---|
| optimizer | Muon (hidden 2D) + AdamW (embeddings) |
| muon_lr | 0.006 |
| adamw_lr | 0.006 |
| lr_schedule | linear_warmdown (warmdown_frac=0.4) |
| batch_size | 80 |
| seq_len | 1024 |
| max_iters | 16000 |
| tokens seen | ~2.79B |
| dataset | FineWeb-Edu-10B |
| best_val_loss | 3.1728 |
Purpose
Interpretability comparison baseline. Trained with identical hyperparameters to
cayley-10b but without the CayleySAE bottleneck at mlp_in. Enables direct
comparison of residual stream representations.
Early stopping (cayley val parity)
Training stopped at the first evaluation where validation CE was at or below 3.173 (published validation loss for aemack-org/cayley-10b). This checkpoint is from iter 4250 (~2.79B tokens seen), not the full max_iters budget in config.json.
- License
- mit
- Created
- Mar 24, 2026
- Updated
- Mar 24, 2026