Downloads · 30 days
0
jwl50-stanford/gdn-grow-coarse64to256-zero-20k-r1
gdn-grow-coarse64to256-zero-20k-r1 is a machine learning model from jwl50-stanford. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch-lightning.
This repository archives the matched zero-padding control from the language-model state-expansion study. It is a PyTorch Lightning training checkpoint, not a directly loadable Transformers AutoModel package.
Downloads · 30 days
0
Access
Public
Updated Jul 22, 2026
Repo size
2.1 GB
Likes
0
Public
Click a slice to open those files.
.ckpt2.1 GB · 100%
From the Hugging Face model README
This repository archives the matched zero-padding control from the language-model state-expansion study. It is a PyTorch Lightning training checkpoint, not a directly loadable Transformers AutoModel package.
m07d11y26-gdn-grow-coarse64to256-zero-20k-r15579022820000last.ckpt2,084,122,635 bytes41a8a6f498abfc7ff2d459dc397c871168e1096b4f5ec5d099ee2b358f69cba41.8.6The archived source file was named step_20000.ckpt; upload uses last.ckpt without changing its bytes.
The model has 12 layers, hidden size 768, 12 heads, maximum Q/K head width 256, V head width 64, sequence length 2048, vocabulary size 50,257, and approximately 173.65M parameters. At token position t, the active Q/K width is
d(t) = 64 + 32 * floor(t / 298), capped at 256.
The six boundaries occur at positions 298, 596, 894, 1192, 1490, and 1788; the mean active width over a length-2048 sequence is approximately 158.2. Newly opened recurrent-state rows are zero padded. There is no learned boundary-completion map in this control.
Training used the Pile, global batch size 256, sequence length 2048, AdamW (lr=5e-4, betas 0.9/0.95, weight decay 0.1), 200 warmup steps, and cosine decay through 20,000 optimizer steps.
The final training log reports validation perplexity 10.30, rounded to two decimals at step 20k. This is a training-run validation-log metric, not a result from the later common L2048 10%-Pile evaluator; no common-slice number is claimed here.
HazyResearch/mlp-mixer@d2b24d685bc12f6e974d3ce68ffe2563a477cea1config/base_experiment.yamlconfig/launch_overrides.yamlconfig/checkpoint_config.yamlprovenance/, with SHA-256 values in provenance/provenance.json.Important qualification: the state-expansion modifications used for this run were preserved as an uncommitted frozen source snapshot; their exact training-time Git commit is unavailable. The bundled files and hashes, rather than the later public branch alone, are the reconstruction source of truth.
Load with torch.load("last.ckpt", map_location="cpu", mmap=True, weights_only=False) in a trusted environment with the matching project, Lightning, and OmegaConf stack. The checkpoint includes optimizer and scheduler state.