Downloads · 30 days
4
14% of all-time downloads
vanilla-v5
markhenry/vanilla-v5
vanilla-v5 is a machine learning model from markhenry. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Vanilla GPT baseline trained to compare against aemack-org/cayley-10b.
All-time downloads
28
Public
Repo size
920 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt920 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Mar 24, 2026
- Updated
- Mar 24, 2026
- SHA
- aa5933d7
Model card
From the Hugging Face model README
vanilla-v5
Vanilla GPT baseline trained to compare against aemack-org/cayley-10b.
Architecture
| Parameter | Value |
|---|---|
| n_layer | 12 |
| n_head | 8 |
| n_embd | 1024 |
| block_size | 1024 |
| vocab_size | 50304 |
| bias | False |
| norm | RMSNorm (affine) |
| MLP | GELU, 4x expansion |
| tokenizer | GPT-2 (tiktoken) |
| dtype | bfloat16 |
| sparsity | none (vanilla) |
Training
| Parameter | Value |
|---|---|
| optimizer | Muon (hidden 2D) + AdamW (embeddings) |
| muon_lr | 0.01 |
| adamw_lr | 0.01 |
| lr_schedule | linear_warmdown (warmdown_frac=0.4) |
| batch_size | 80 |
| seq_len | 1024 |
| max_iters | 16000 |
| tokens seen | ~1.3B |
| dataset | FineWeb-Edu-10B |
| best_val_loss | 2.9109 |
Purpose
Interpretability comparison baseline. Trained with identical hyperparameters to
cayley-10b but without the CayleySAE bottleneck at mlp_in. Enables direct
comparison of residual stream representations.
- License
- mit
- Created
- Mar 24, 2026
- Updated
- Mar 24, 2026