Downloads · 30 days
0
jwl50-stanford/mom-gdn-120m-m2-softmax
mom-gdn-120m-m2-softmax is a machine learning model from jwl50-stanford. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository contains the exact optimizer-step-20,000 PyTorch Lightning checkpoint for the residual banked Gated DeltaNet architecture in HazyResearch/mixture-of-memories-dev. Writes use a dense softmax router. Rea…
Downloads · 30 days
0
Access
Public
Updated Aug 22, 2026
Repo size
1.6 GB
Likes
0
Public
Click a slice to open those files.
.ckpt1.6 GB · 100%
From the Hugging Face model README
This repository contains the exact optimizer-step-20,000 PyTorch Lightning
checkpoint for the residual banked Gated DeltaNet architecture in
HazyResearch/mixture-of-memories-dev.
Writes use a dense softmax router. Reads use softmax gates
over 2 persistent memory banks.
Historical softmax-read M2; this is not an arm of the matched independent-softplus M2/M4/M8 capacity sweep.
step_20000.ckpt371516a267c3be0818577900fe451bbef2b62bf49b4b7b6eaeec6dcd078fcdc0b5edec8db283704f1180be089d26507dd95107dbours/residual_gdn_m2_softmaxlanguage_modeling/gdn_mom_m2_residual_softmax_120m_20kThe file is a full Lightning checkpoint, including trainer/optimizer state. It
is published in that format because the project evaluator loads exact .ckpt
artifacts directly.
On the established project protocols, validation PPL is
10.0111, rare first-recall PPL is
3.6467, and FDA exact-match accuracy is
7.35%. See the source repository for cohort
and protocol definitions.
After installing the source repository, use its checked-in model source YAML:
mom-eval \
--model-yaml configs/evaluation/source/mom_m2_softmax_120m_20k.yaml \
--pile-data-dir /path/to/pile \
--suite standard
The YAML pins both this repository revision and the checkpoint SHA-256.