Downloads · 30 days
0
jwl50-stanford/mom-gdn-120m-m4-softplus
mom-gdn-120m-m4-softplus is a machine learning model from jwl50-stanford. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository contains the exact optimizer-step-20,000 PyTorch Lightning checkpoint for the residual banked Gated DeltaNet architecture in HazyResearch/mixture-of-memories-dev. Writes use a dense softmax router. Rea…
Downloads · 30 days
0
Access
Public
Updated Aug 22, 2026
Repo size
1.6 GB
Likes
0
Public
Click a slice to open those files.
.ckpt1.6 GB · 100%
From the Hugging Face model README
This repository contains the exact optimizer-step-20,000 PyTorch Lightning
checkpoint for the residual banked Gated DeltaNet architecture in
HazyResearch/mixture-of-memories-dev.
Writes use a dense softmax router. Reads use independent_softplus gates
over 4 persistent memory banks.
Matched independent-softplus capacity release; compare directly with M8.
step_20000.ckpt8e6d21cc4402c1c438041ae45b64cf17abc3b8f6f43d6f77fafa5c0df3db0dbb977c43624e7d70bbf2a89b0813e15077005a265aours/residual_gdn_m4_softpluslanguage_modeling/gdn_mom_m4_residual_softplus_120m_20kThe file is a full Lightning checkpoint, including trainer/optimizer state. It
is published in that format because the project evaluator loads exact .ckpt
artifacts directly.
On the established project protocols, validation PPL is
9.9497, rare first-recall PPL is
3.5431, and FDA exact-match accuracy is
10.34%. See the source repository for cohort
and protocol definitions.
After installing the source repository, use its checked-in model source YAML:
mom-eval \
--model-yaml configs/evaluation/source/mom_m4_softplus_120m_20k.yaml \
--pile-data-dir /path/to/pile \
--suite standard
The YAML pins both this repository revision and the checkpoint SHA-256.