Downloads · 30 days
390
63% of all-time downloads
GLA-340M
linear-moe-hub/GLA-340M
GLA-340M is a machine learning model from linear-moe-hub. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Model of the paper MoM: Linear Sequence Modeling with Mixture-of-Memories and Gated Linear Attention Transformers with Hardware-Efficient Training.
All-time downloads
620
Public
Parameters
374M
749 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors749 MB · 99%
At a glance
- License
- apache-2.0
- Model type
- gla
- Access
- Public
- Created
- Mar 18, 2025
- Updated
- Mar 18, 2025
- SHA
- 81034cc3
Datasets
Model card
From the Hugging Face model README
Model of the paper MoM: Linear Sequence Modeling with Mixture-of-Memories and Gated Linear Attention Transformers with Hardware-Efficient Training.
The model was trained on a sample of SlimPajama with 15B tokens.
Due to changes in the MLP layer structure in the latest version of fla, the weights cannot be loaded. You can use the version at fla instead.
- Type
- gla
- License
- apache-2.0
- Languages
- en
- Created
- Mar 18, 2025
- Updated
- Mar 18, 2025