Downloads · 30 days
0
gdn-120m-hdim128-v64-20k
robgarct/gdn-120m-hdim128-v64-20k
gdn-120m-hdim128-v64-20k is a machine learning model from robgarct. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for recurrent-recall-circuits.
120M gated DeltaNet (headk 128, headv 64) language model, 20k steps of Pile pretraining, published so its recall evaluations can be reproduced.
Access
Public
Updated Sep 8, 2026
Repo size
581 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.ckpt581 MB · 100%
At a glance
- Library
- recurrent-recall-circuits
- Access
- Public
- Created
- Sep 8, 2026
- Updated
- Sep 8, 2026
- SHA
- ebce7655
Model card
From the Hugging Face model README
gdn-120m-hdim128-v64-20k
120M gated DeltaNet (head_k 128, head_v 64) language model, 20k steps of Pile pretraining, published so its
recall evaluations can be reproduced.
- trained from experiment config:
lm_120m/gdn_paper_hdim128_expandv0.5 - checkpoint file:
final.ckpt(weights only; optimizer state stripped) global_step20000
Part of a state-width sweep evaluated on FDA/SWDE/SQuAD, RULER NIAH, and a distance-controlled retention sweep.
- Library
- recurrent-recall-circuits
- Created
- Sep 8, 2026
- Updated
- Sep 8, 2026