vanilla-large-20B

vanilla-large-20B — dense baselines (24L / d=2048, 1.31B)

Dense (sparsity_mode=none) counterparts to the ParityTransformer large-chassis flagship markhenry/cayley-large-2L-mlp_in-20B. All checkpoints come from one 20B-token run, so both baseline definitions share a single trajectory.

Checkpoints

Files are named by iter. (They were previously named by the --save-at-val threshold that fired; those names misstated the actual loss — ckpt_val_2.8000.pt held a model at 2.7950 — so they were renamed. Two thresholds fired on the same eval and produced byte-identical files; only one copy is kept.)

Reference, measured in the same pass on the same batches:

ckpt_val_3400.pt fired a --save-at-val trigger but is 0.0023 nats worse than the flagship (7σ) — it does not clear it. It does clear the aux-loss variant by 0.0328 (86σ). ckpt_val_3800.pt clears both.

ckpt_val_3800.pt being better than the flagship is the useful direction: a dense model that beats the ParityTransformer on val and is still less interpretable cannot be dismissed as a handicapped baseline.

ckpt_best.pt (iter 12400) is not uploaded — ckpt_final_20B.pt measures better, so the "best" label was a noise artifact of the 3.28M-token in-run eval and the file carries no information the final one lacks.

How these numbers were measured

evals/language_modeling/eval_val_loss.py over the entire FineWeb-Edu val split — contiguous non-overlapping windows, bf16 autocast, batch 8 × 1024, --data fineweb-edu-25B, seed 0.

The paired SEM is ~17× tighter because shared passage difficulty cancels. Quote the paired differences, not error bars added in quadrature.

Two reasons the run log disagrees, both of which bite if you mix sources:

The flagship's published 2.8081 is an in-run number; on this protocol it is 2.7957. Use 2.7957 against these checkpoints and do not mix the two.

Alignment tax

Equal budget (20B tokens both sides):

Token efficiency — where dense first reaches the flagship's measured val, interpolated between the two bracketing measured points:

Caveats before this goes in a paper table:

Recipe: identical to the flagship except sparsity_mode

Deltas vs the flagship:

Loading

import torch
ck = torch.load("ckpt_val_3800.pt", map_location="cpu", weights_only=False)
ck["iter_num"]                              # 3800
cfg, sd = ck["model_config"], ck["model"]   # GPTConfig fields, 148 tensors

Checkpoints carry optimizer_states (Muon momentum + AdamW), so they are resumable, not inference-only.

Siblings

• markhenry/cayley-large-2L-mlp_in-20B — the flagship this is matched to
• markhenry/vanilla-small-20B — same construction, small chassis

---

Measurements and card by Claude Code, from the run log and a paired re-evaluation of every checkpoint over the whole val split.