Downloads · 30 days
0
jwl50-stanford/gdn-paper-120m-fixed-d256-fp32res-gpt2init-20k-r1
gdn-paper-120m-fixed-d256-fp32res-gpt2init-20k-r1 is a machine learning model from jwl50-stanford. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch-lightning.
This repository archives the canonical fixed-width Gated DeltaNet (GDN) language model with Q/K head dimension 256 and V head dimension 64. It is a PyTorch Lightning training checkpoint, not a directly loadable Transf…
Downloads · 30 days
0
Access
Public
Updated Jul 22, 2026
Repo size
2.1 GB
Likes
0
Public
Click a slice to open those files.
.ckpt2.1 GB · 100%
From the Hugging Face model README
This repository archives the canonical fixed-width Gated DeltaNet (GDN) language model with Q/K head dimension 256 and V head dimension 64. It is a PyTorch Lightning training checkpoint, not a directly loadable Transformers AutoModel package. Reconstruct the model with the bundled Hydra experiment config and the referenced HazyResearch/mlp-mixer code.
m05d15y26-gdn-paper-d256-fp32res-gpt2init-20k-r15302469820000last.ckpt2,084,104,035 bytesf42ba43f80a26aa991d5441fbfa3cab1113e5648df50f286c3d473d1dd016ed11.8.6The archived source file was named step_20000.ckpt; upload uses last.ckpt without changing its bytes.
The model has 12 layers, hidden size 768, 12 Q/K/V heads, fixed Q/K head dimension 256, V head dimension 64 (expand_v=0.25), sequence length 2048, vocabulary size 50,257, and approximately 173.65M parameters. It uses an fp32 residual stream and GPT-2-style residual-output initialization.
Training used the Pile, global batch size 256, AdamW (lr=5e-4, betas 0.9/0.95, weight decay 0.1), 200 warmup steps, and cosine decay to 5e-5 through 20,000 optimizer steps. The exact experiment YAML is in config/experiment.yaml.
| Evaluation | Perplexity | Qualification |
|---|---|---|
| Final training-run validation log | 9.980 | Rounded display in the step-20k training log; it is not the common-slice evaluator. |
| Common L2048 Pile evaluation | 9.87802532382 | Independent shared 10% validation-slice protocol: 18,720 sequences / 38,338,560 tokens. |
These values use different evaluation pipelines and should not be treated as repeated estimates from one identical sample.
HazyResearch/mlp-mixer@d2b24d685bc12f6e974d3ce68ffe2563a477cea1train/configs/experiment/mlp_mixer/gdn_paper_hdim256_fp32res_gpt2init_20k.yamlLoad with torch.load("last.ckpt", map_location="cpu", mmap=True, weights_only=False) in a trusted environment with the matching Lightning/OmegaConf stack, then restore through the project Lightning module. The file contains optimizer and scheduler state as well as model weights.