Downloads · 30 days
0
jwl50-stanford/mom-gdn-120m-m8-softplus
mom-gdn-120m-m8-softplus is a machine learning model from jwl50-stanford. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository contains the exact optimizer-step-20,000 PyTorch Lightning checkpoint for the residual banked Gated DeltaNet architecture in HazyResearch/mixture-of-memories-dev. Writes use a dense softmax router. Rea…
Downloads · 30 days
0
Access
Public
Updated Aug 22, 2026
Repo size
1.6 GB
Likes
0
Public
Click a slice to open those files.
.ckpt1.6 GB · 100%
From the Hugging Face model README
This repository contains the exact optimizer-step-20,000 PyTorch Lightning
checkpoint for the residual banked Gated DeltaNet architecture in
HazyResearch/mixture-of-memories-dev.
Writes use a dense softmax router. Reads use independent_softplus gates
over 8 persistent memory banks.
Matched independent-softplus capacity release; compare directly with M4.
step_20000.ckpt51db7a9541ef82429ad106a6e06f505681d4d5159d6346275a45508be7bfa16a8ef1804f015b2ddef9b623eda1906b1fe497b21aours/residual_gdn_m8_softpluslanguage_modeling/gdn_mom_m8_residual_softplus_120m_20kThe file is a full Lightning checkpoint, including trainer/optimizer state. It
is published in that format because the project evaluator loads exact .ckpt
artifacts directly.
On the established project protocols, validation PPL is
9.913, rare first-recall PPL is
3.389, and FDA exact-match accuracy is
12.89%. See the source repository for cohort
and protocol definitions.
After installing the source repository, use its checked-in model source YAML:
mom-eval \
--model-yaml configs/evaluation/source/mom_m8_softplus_120m_20k.yaml \
--pile-data-dir /path/to/pile \
--suite standard
The YAML pins both this repository revision and the checkpoint SHA-256.